2024-03-23T14:24:45.354 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T14:24:45.354 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384 --verbose --owner scheduled_pdonnell@teuthology --description fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} --name pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi -- /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/orig.config.yaml 2024-03-23T14:24:45.399 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T14:24:45.494 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384 branch: wip-batrick-testing-20240323.003144-squid description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} email: pdonnell@redhat.com first_in_suite: false job_id: '7618384' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-batrick-testing-20240323.003144-squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 pid file: '' rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability sha1: 21b3bd672163092b4e542f772797fcba02d3d4de ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 21b3bd672163092b4e542f772797fcba02d3d4de selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 subvolume_version: 1 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-batrick-testing-20240323.003144-squid sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b owner: scheduled_pdonnell@teuthology priority: 85 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - client.2 - - mon.b - mon.c - mgr.x - mds.b - mds.d - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - client.3 seed: 4284 sha1: 21b3bd672163092b4e542f772797fcba02d3d4de sleep_before_teardown: 0 subset: 405/512 suite: fs suite_branch: wip-batrick-testing-20240323.003144-squid suite_path: /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa suite_relpath: qa suite_repo: https://github.com/batrick/ceph.git suite_sha1: d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi161.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKdulghjUx2dBVzfkJOysFyenP9PDwo02ofmLKEojk9EStLnapx7pnDP8GFdWOec+snZKV2BIzwqhOjM9JWDjM= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC7RNbUBA5eX59Y7zf4jpLUGwBV7jKzulhCtUEFQOIZwAED998+oGq2kxB7gy9IynqcatepIhLGMJoYIJxvDj0= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_pool_perm teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-23_09:56:53 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-23T14:24:45.494 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa; will attempt to use it 2024-03-23T14:24:45.495 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks 2024-03-23T14:24:45.495 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T14:24:45.497 INFO:teuthology.task.internal:Checking packages... 2024-03-23T14:24:45.519 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '21b3bd672163092b4e542f772797fcba02d3d4de' 2024-03-23T14:24:45.519 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T14:24:45.519 INFO:teuthology.packaging:ref: None 2024-03-23T14:24:45.519 INFO:teuthology.packaging:tag: None 2024-03-23T14:24:45.519 INFO:teuthology.packaging:branch: wip-batrick-testing-20240323.003144-squid 2024-03-23T14:24:45.519 INFO:teuthology.packaging:sha1: 21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T14:24:45.520 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-batrick-testing-20240323.003144-squid 2024-03-23T14:24:45.742 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1855.g21b3bd67 2024-03-23T14:24:45.743 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T14:24:45.749 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T14:24:45.749 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T14:24:45.790 INFO:teuthology.task.internal:Saving configuration 2024-03-23T14:24:45.806 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T14:24:45.811 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T14:24:45.837 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 14:19:37.511253', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKdulghjUx2dBVzfkJOysFyenP9PDwo02ofmLKEojk9EStLnapx7pnDP8GFdWOec+snZKV2BIzwqhOjM9JWDjM='} 2024-03-23T14:24:45.860 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 14:19:37.509919', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC7RNbUBA5eX59Y7zf4jpLUGwBV7jKzulhCtUEFQOIZwAED998+oGq2kxB7gy9IynqcatepIhLGMJoYIJxvDj0='} 2024-03-23T14:24:45.860 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T14:24:45.866 INFO:teuthology.task.internal:roles: ubuntu@smithi161.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'] 2024-03-23T14:24:45.866 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3'] 2024-03-23T14:24:45.866 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T14:24:45.958 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f90fd3b1160>, signals=[15]) 2024-03-23T14:24:45.959 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T14:24:45.967 INFO:teuthology.task.internal:Opening connections... 2024-03-23T14:24:45.967 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:24:45.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:24:46.062 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:24:46.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:24:46.153 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T14:24:46.160 DEBUG:teuthology.orchestra.run.smithi161:> uname -m 2024-03-23T14:24:46.179 INFO:teuthology.orchestra.run.smithi161.stdout:x86_64 2024-03-23T14:24:46.179 DEBUG:teuthology.orchestra.run.smithi161:> cat /etc/os-release 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:NAME="CentOS Stream" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION="9" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:ID="centos" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:ID_LIKE="rhel fedora" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:VERSION_ID="9" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:PLATFORM_ID="platform:el9" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:ANSI_COLOR="0;31" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:LOGO="fedora-logo-icon" 2024-03-23T14:24:46.236 INFO:teuthology.orchestra.run.smithi161.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T14:24:46.237 INFO:teuthology.orchestra.run.smithi161.stdout:HOME_URL="https://centos.org/" 2024-03-23T14:24:46.237 INFO:teuthology.orchestra.run.smithi161.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T14:24:46.237 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T14:24:46.237 INFO:teuthology.orchestra.run.smithi161.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T14:24:46.237 INFO:teuthology.lock.ops:Updating smithi161.front.sepia.ceph.com on lock server 2024-03-23T14:24:46.259 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-23T14:24:46.277 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-23T14:24:46.277 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-23T14:24:46.335 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T14:24:46.336 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T14:24:46.336 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T14:24:46.336 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-23T14:24:46.367 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T14:24:46.375 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T14:24:46.391 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T14:24:46.391 DEBUG:teuthology.orchestra.run.smithi161:> test '!' -e /home/ubuntu/cephtest 2024-03-23T14:24:46.394 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-23T14:24:46.410 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T14:24:46.416 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T14:24:46.416 DEBUG:teuthology.orchestra.run.smithi161:> test -z $(ls -A /var/lib/ceph) 2024-03-23T14:24:46.450 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-23T14:24:46.542 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T14:24:46.597 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T14:24:46.610 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-23T14:24:46.611 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-23T14:24:46.611 DEBUG:teuthology.orchestra.run.smithi161:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T14:24:46.611 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T14:24:46.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:24:46.627 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-23T14:24:46.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:24:46.628 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-03-23T14:24:46.684 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-23T14:24:46.685 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-23T14:24:46.685 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-23T14:24:46.686 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-361.el9.x86_64 2024-03-23T14:24:46.687 INFO:teuthology.task.kernel:Running kernel on smithi161: 5.14.0-361.el9.x86_64 2024-03-23T14:24:46.687 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum install -y kernel 2024-03-23T14:24:48.848 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-23T14:24:48.957 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-23T14:24:50.866 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 2.5 MB/s | 2.0 MB 00:00 2024-03-23T14:24:50.916 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 2.1 MB/s | 2.0 MB 00:00 2024-03-23T14:24:51.814 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-23T14:24:52.430 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 6.7 MB/s | 8.0 MB 00:01 2024-03-23T14:24:54.098 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 30 MB/s | 19 MB 00:00 2024-03-23T14:24:54.907 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 9.3 MB/s | 19 MB 00:02 2024-03-23T14:24:58.916 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-23T14:24:59.457 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-23T14:25:00.700 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 24 kB/s | 16 kB 00:00 2024-03-23T14:25:00.766 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 100 kB/s | 16 kB 00:00 2024-03-23T14:25:02.767 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-23T14:25:04.060 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 6.4 MB/s | 21 MB 00:03 2024-03-23T14:25:07.775 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-03-23T14:25:08.959 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-23T14:25:09.840 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-23T14:25:09.840 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-23T14:25:09.934 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-23T14:25:09.944 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-23T14:25:09.945 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:25:09.946 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 76 M 2024-03-23T14:25:09.946 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-23T14:25:09.946 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-23T14:25:10.143 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 696 kB/s | 44 kB 00:00 2024-03-23T14:25:10.885 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 18 MB/s | 15 MB 00:00 2024-03-23T14:25:11.066 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-23T14:25:11.066 INFO:teuthology.orchestra.run.smithi161.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-23T14:25:11.160 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-23T14:25:11.165 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout: Package Architecture Version Repository Size 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout:================================================================================ 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout:Install 4 Packages 2024-03-23T14:25:11.166 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:25:11.167 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 76 M 2024-03-23T14:25:11.167 INFO:teuthology.orchestra.run.smithi161.stdout:Installed size: 123 M 2024-03-23T14:25:11.167 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-23T14:25:11.328 INFO:teuthology.orchestra.run.smithi161.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 460 kB/s | 44 kB 00:00 2024-03-23T14:25:11.435 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 21 MB/s | 27 MB 00:01 2024-03-23T14:25:11.720 INFO:teuthology.orchestra.run.smithi161.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 31 MB/s | 15 MB 00:00 2024-03-23T14:25:11.727 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 20 MB/s | 33 MB 00:01 2024-03-23T14:25:11.728 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-23T14:25:11.728 INFO:teuthology.orchestra.run.smithi183.stdout:Total 43 MB/s | 76 MB 00:01 2024-03-23T14:25:12.103 INFO:teuthology.orchestra.run.smithi161.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 35 MB/s | 27 MB 00:00 2024-03-23T14:25:12.395 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-23T14:25:12.428 INFO:teuthology.orchestra.run.smithi161.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-03-23T14:25:12.428 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-23T14:25:12.429 INFO:teuthology.orchestra.run.smithi161.stdout:Total 60 MB/s | 76 MB 00:01 2024-03-23T14:25:12.765 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-23T14:25:12.765 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-23T14:25:13.098 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-23T14:25:13.448 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-23T14:25:13.448 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-23T14:25:13.778 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-23T14:25:13.778 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-23T14:25:14.447 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-23T14:25:14.447 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-23T14:25:16.257 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-23T14:25:16.847 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-23T14:25:16.939 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-23T14:25:16.943 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T14:25:17.542 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-23T14:25:17.637 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T14:25:19.672 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T14:25:19.805 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T14:25:20.404 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T14:25:20.549 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T14:25:24.506 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T14:25:24.554 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:25.238 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T14:25:25.357 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:29.310 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:30.107 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:58.401 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T14:25:58.401 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:25:58.445 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:58.785 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:58.785 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-23T14:25:58.785 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T14:25:58.785 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T14:25:59.194 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:59.195 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:25:59.195 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-23T14:25:59.195 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-23T14:25:59.195 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-23T14:25:59.195 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-23T14:25:59.195 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-23T14:25:59.195 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:25:59.195 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-23T14:25:59.513 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-23T14:25:59.541 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:59.541 INFO:teuthology.orchestra.run.smithi161.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T14:25:59.541 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:25:59.585 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:59.608 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-23T14:25:59.609 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-23T14:25:59.610 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-23T14:25:59.610 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-23T14:25:59.610 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-23T14:25:59.610 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-23T14:25:59.610 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-23T14:25:59.610 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-23T14:25:59.690 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-23T14:25:59.690 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-23T14:25:59.690 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-23T14:25:59.724 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-23T14:25:59.725 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:25:59.726 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T14:25:59.726 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T14:25:59.727 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-430.el9.x86_64 2024-03-23T14:25:59.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T14:25:59.990 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:25:59.990 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-23T14:25:59.990 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-23T14:25:59.990 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-23T14:26:00.309 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:26:00.417 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-23T14:26:00.749 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-03-23T14:26:00.869 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-23T14:26:00.870 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-23T14:26:00.870 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-23T14:26:00.870 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-23T14:26:00.870 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-23T14:26:00.870 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-23T14:26:00.870 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi161.front.sepia.ceph.com, path=None, version=distro) 2024-03-23T14:26:00.870 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q kernel | sort -rV | head -n 1 2024-03-23T14:26:00.949 INFO:teuthology.orchestra.run.smithi161.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-23T14:26:00.949 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-23T14:26:00.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo rpm -qi grub2-tools 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Name : grub2-tools 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Epoch : 1 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Version : 2.06 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Release : 68.el9 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Architecture: x86_64 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Group : Unspecified 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Size : 8264254 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:License : GPLv3+ 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T14:26:00.984 INFO:teuthology.orchestra.run.smithi161.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:Packager : builder@centos.org 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:Vendor : CentOS 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:Summary : Support tools for GRUB. 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:Description : 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:hardware devices. 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:26:00.985 INFO:teuthology.orchestra.run.smithi161.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T14:26:00.986 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T14:26:00.986 INFO:teuthology.task.kernel:Updating grub on smithi161 to boot 5.14.0-430.el9.x86_64 2024-03-23T14:26:00.987 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T14:26:01.632 INFO:teuthology.orchestra.run.smithi161.stderr:Generating grub configuration file ... 2024-03-23T14:26:01.694 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T14:26:01.727 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-23T14:26:01.729 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-23T14:26:01.744 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.IIOmVoKCxM 2024-03-23T14:26:01.745 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IIOmVoKCxM 2024-03-23T14:26:01.813 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.IIOmVoKCxM 2024-03-23T14:26:01.962 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.IIOmVoKCxM is 6KB 2024-03-23T14:26:02.010 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.IIOmVoKCxM 2024-03-23T14:26:02.023 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T14:26:02.088 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T14:26:02.088 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T14:26:02.088 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T14:26:02.088 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-23T14:26:02.089 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-23T14:26:02.242 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-23T14:26:03.002 INFO:teuthology.orchestra.run.smithi161.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T14:26:03.039 INFO:teuthology.orchestra.run.smithi161.stderr:done 2024-03-23T14:26:03.041 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-03-23T14:26:03.057 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.yTI3is2yXO 2024-03-23T14:26:03.057 DEBUG:teuthology.orchestra.run.smithi161:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yTI3is2yXO 2024-03-23T14:26:03.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.yTI3is2yXO 2024-03-23T14:26:03.265 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.yTI3is2yXO is 6KB 2024-03-23T14:26:03.314 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.yTI3is2yXO 2024-03-23T14:26:03.329 DEBUG:teuthology.orchestra.run.smithi161:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T14:26:03.393 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T14:26:03.394 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T14:26:03.394 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T14:26:03.394 INFO:teuthology.orchestra.run.smithi161.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-23T14:26:03.395 DEBUG:teuthology.orchestra.run.smithi161:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-23T14:26:03.555 DEBUG:teuthology.orchestra.run.smithi161:> sudo shutdown -r now 2024-03-23T14:26:32.269 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-23T14:26:32.270 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-23T14:26:32.271 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:26:33.560 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-23T14:26:33.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-23T14:26:33.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:26:50.783 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-23T14:26:52.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-03-23T14:26:59.790 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-23T14:26:59.791 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:27:01.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-23T14:27:01.064 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:27:02.846 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-23T14:27:04.126 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-03-23T14:27:14.858 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-23T14:27:14.859 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:27:16.128 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-23T14:27:16.129 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:27:34.559 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.161 2024-03-23T14:27:46.749 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-23T14:27:47.165 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-23T14:27:47.165 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-23T14:27:47.165 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-23T14:27:47.179 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-430.el9.x86_64 2024-03-23T14:27:47.179 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-23T14:27:47.179 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T14:27:47.179 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T14:27:48.181 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-23T14:27:48.181 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T14:27:48.260 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-23T14:27:48.299 DEBUG:teuthology.parallel:result is None 2024-03-23T14:27:49.560 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-23T14:27:49.561 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:27:49.908 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-03-23T14:27:50.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-23T14:27:50.489 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-23T14:27:50.489 DEBUG:teuthology.orchestra.run.smithi161:> uname -r 2024-03-23T14:27:50.504 INFO:teuthology.orchestra.run.smithi161.stdout:5.14.0-430.el9.x86_64 2024-03-23T14:27:50.504 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-23T14:27:50.505 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T14:27:50.505 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T14:27:51.505 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-23T14:27:51.505 DEBUG:teuthology.orchestra.run.smithi161:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-23T14:27:51.618 INFO:teuthology.orchestra.run.smithi161.stdout:ttyS1 2024-03-23T14:27:51.652 DEBUG:teuthology.parallel:result is None 2024-03-23T14:27:51.653 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-23T14:27:51.663 INFO:teuthology.task.internal:Creating test directory... 2024-03-23T14:27:51.663 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T14:27:51.665 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T14:27:51.683 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-23T14:27:51.702 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-23T14:27:51.736 INFO:teuthology.task.internal:Creating archive directory... 2024-03-23T14:27:51.736 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T14:27:51.738 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T14:27:51.780 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-23T14:27:51.787 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-23T14:27:51.787 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T14:27:51.796 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T14:27:51.834 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T14:27:51.847 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T14:27:51.852 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T14:27:51.865 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T14:27:51.867 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-23T14:27:51.874 INFO:teuthology.task.internal:Configuring sudo... 2024-03-23T14:27:51.874 DEBUG:teuthology.orchestra.run.smithi161:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T14:27:51.891 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T14:27:51.936 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-23T14:27:51.945 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-23T14:27:51.945 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T14:27:51.957 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T14:27:51.993 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T14:27:52.052 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T14:27:52.099 DEBUG:teuthology.orchestra.run.smithi161:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T14:27:52.179 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T14:27:52.222 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:27:52.222 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T14:27:52.290 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T14:27:52.336 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T14:27:52.391 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T14:27:52.475 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T14:27:52.518 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:27:52.518 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T14:27:52.586 DEBUG:teuthology.orchestra.run.smithi161:> sudo service rsyslog restart 2024-03-23T14:27:52.588 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-23T14:27:52.636 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T14:27:52.675 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T14:27:53.143 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-23T14:27:53.151 INFO:teuthology.task.internal:Starting timer... 2024-03-23T14:27:53.151 INFO:teuthology.run_tasks:Running task pcp... 2024-03-23T14:27:53.198 INFO:teuthology.run_tasks:Running task selinux... 2024-03-23T14:27:53.230 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-03-23T14:27:53.231 DEBUG:teuthology.orchestra.run.smithi161:> sudo service auditd rotate 2024-03-23T14:27:53.290 INFO:teuthology.orchestra.run.smithi161.stdout:Rotating logs: 2024-03-23T14:27:53.292 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-23T14:27:53.343 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-23T14:27:53.345 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-23T14:27:53.345 DEBUG:teuthology.orchestra.run.smithi161:> /usr/sbin/getenforce 2024-03-23T14:27:53.368 INFO:teuthology.orchestra.run.smithi161.stdout:Permissive 2024-03-23T14:27:53.368 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-23T14:27:53.409 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-23T14:27:53.410 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi161.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-23T14:27:53.410 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T14:27:53.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:27:53.437 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T14:27:53.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:27:53.478 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-23T14:27:53.478 DEBUG:teuthology.orchestra.run.smithi161:> sudo /usr/sbin/setenforce permissive 2024-03-23T14:27:53.510 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-23T14:27:53.553 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-23T14:27:53.564 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-23T14:27:53.579 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-23T14:27:53.581 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi161.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-23T14:32:06.863 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi161.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-23T14:32:06.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-23T14:32:06.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi161.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:32:06.960 DEBUG:teuthology.orchestra.run.smithi161:> true 2024-03-23T14:32:07.049 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi161.front.sepia.ceph.com' 2024-03-23T14:32:07.049 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-23T14:32:07.050 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T14:32:07.140 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-23T14:32:07.219 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-23T14:32:07.219 INFO:teuthology.run_tasks:Running task clock... 2024-03-23T14:32:07.229 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-23T14:32:07.230 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T14:32:07.230 DEBUG:teuthology.orchestra.run.smithi161:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T14:32:07.232 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-23T14:32:07.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T14:32:07.261 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T14:32:07.278 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T14:32:07.293 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-23T14:32:07.308 INFO:teuthology.orchestra.run.smithi161.stderr:sudo: ntpd: command not found 2024-03-23T14:32:07.311 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-23T14:32:07.321 INFO:teuthology.orchestra.run.smithi161.stdout:506 Cannot talk to daemon 2024-03-23T14:32:07.338 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T14:32:07.341 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-23T14:32:07.354 INFO:teuthology.orchestra.run.smithi161.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T14:32:07.355 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-23T14:32:07.373 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-23T14:32:07.389 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-23T14:32:07.410 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-03-23T14:32:07.414 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T14:32:07.414 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-03-23T14:32:07.414 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:32:07.414 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:32:07.414 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:32:07.414 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:32:07.449 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-23T14:32:07.452 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T14:32:07.452 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-23T14:32:07.452 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:32:07.452 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:32:07.452 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:32:07.452 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:32:07.453 INFO:teuthology.run_tasks:Running task install... 2024-03-23T14:32:07.461 DEBUG:teuthology.task.install:project ceph 2024-03-23T14:32:07.462 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}} 2024-03-23T14:32:07.462 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'} 2024-03-23T14:32:07.462 INFO:teuthology.task.install:Using flavor: default 2024-03-23T14:32:07.467 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-23T14:32:07.468 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-23T14:32:07.468 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-23T14:32:07.468 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T14:32:07.470 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'tag': None, 'wait_for_package': False} 2024-03-23T14:32:07.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T14:32:07.701 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-23T14:32:07.702 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-23T14:32:07.705 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/ 2024-03-23T14:32:07.705 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1855.g21b3bd67 2024-03-23T14:32:07.814 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T14:32:07.814 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:32:07.814 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T14:32:07.822 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-batrick-testing-20240323.003144-squid/21b3bd672163092b4e542f772797fcba02d3d4de/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-23T14:32:07.822 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:32:07.822 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-23T14:32:07.843 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T14:32:07.843 DEBUG:teuthology.orchestra.run.smithi161:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T14:32:07.852 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-23T14:32:07.852 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/21b3bd672163092b4e542f772797fcba02d3d4de/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-23T14:32:07.913 DEBUG:teuthology.orchestra.run.smithi161:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T14:32:07.922 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-23T14:32:07.991 DEBUG:teuthology.orchestra.run.smithi161:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T14:32:08.003 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-23T14:32:08.058 INFO:teuthology.orchestra.run.smithi161.stdout:check_obsoletes = 1 2024-03-23T14:32:08.059 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum clean all 2024-03-23T14:32:08.070 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-23T14:32:08.072 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-23T14:32:08.301 INFO:teuthology.orchestra.run.smithi161.stdout:74 files removed 2024-03-23T14:32:08.307 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-23T14:32:08.327 DEBUG:teuthology.orchestra.run.smithi161:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T14:32:08.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-23T14:32:08.958 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 256 kB/s | 83 kB 00:00 2024-03-23T14:32:08.993 INFO:teuthology.orchestra.run.smithi161.stdout:ceph packages for x86_64 227 kB/s | 83 kB 00:00 2024-03-23T14:32:09.367 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 28 kB/s | 10 kB 00:00 2024-03-23T14:32:09.377 INFO:teuthology.orchestra.run.smithi161.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-23T14:32:09.701 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 6.4 kB/s | 1.9 kB 00:00 2024-03-23T14:32:09.786 INFO:teuthology.orchestra.run.smithi161.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-23T14:32:10.154 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-23T14:32:10.263 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-23T14:32:11.637 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 5.4 MB/s | 2.0 MB 00:00 2024-03-23T14:32:11.730 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS-9 - BaseOS 6.3 MB/s | 2.0 MB 00:00 2024-03-23T14:32:12.730 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-23T14:32:14.046 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.0 MB 00:01 2024-03-23T14:32:14.620 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-23T14:32:15.830 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - AppStream 25 MB/s | 19 MB 00:00 2024-03-23T14:32:18.621 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 12 MB/s | 6.1 MB 00:00 2024-03-23T14:32:20.247 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-23T14:32:20.448 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - CRB 5.7 MB/s | 6.1 MB 00:01 2024-03-23T14:32:21.815 INFO:teuthology.orchestra.run.smithi161.stdout:CentOS Stream 9 - Extras packages 84 kB/s | 16 kB 00:00 2024-03-23T14:32:22.783 INFO:teuthology.orchestra.run.smithi161.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-23T14:32:24.267 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 21 MB 00:03 2024-03-23T14:32:27.708 INFO:teuthology.orchestra.run.smithi161.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-03-23T14:32:29.139 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-23T14:32:30.077 INFO:teuthology.orchestra.run.smithi161.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T14:32:30.078 INFO:teuthology.orchestra.run.smithi161.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T14:32:30.090 INFO:teuthology.orchestra.run.smithi161.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T14:32:30.091 INFO:teuthology.orchestra.run.smithi161.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T14:32:30.092 INFO:teuthology.orchestra.run.smithi161.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T14:32:30.093 INFO:teuthology.orchestra.run.smithi161.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T14:32:30.095 INFO:teuthology.orchestra.run.smithi161.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T14:32:30.097 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T14:32:30.098 INFO:teuthology.orchestra.run.smithi161.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T14:32:30.180 INFO:teuthology.orchestra.run.smithi161.stdout:Dependencies resolved. 2024-03-23T14:32:30.193 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-23T14:32:30.193 INFO:teuthology.orchestra.run.smithi161.stdout: Package Arch Version Repository Size 2024-03-23T14:32:30.193 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-23T14:32:30.193 INFO:teuthology.orchestra.run.smithi161.stdout:Installing: 2024-03-23T14:32:30.193 INFO:teuthology.orchestra.run.smithi161.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T14:32:30.193 INFO:teuthology.orchestra.run.smithi161.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-23T14:32:30.193 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-23T14:32:30.193 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-23T14:32:30.194 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T14:32:30.195 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrading: 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T14:32:30.196 INFO:teuthology.orchestra.run.smithi161.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T14:32:30.197 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout:Installing dependencies: 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-23T14:32:30.198 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-23T14:32:30.199 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-23T14:32:30.200 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-23T14:32:30.201 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T14:32:30.202 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T14:32:30.203 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T14:32:30.204 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-23T14:32:30.205 INFO:teuthology.orchestra.run.smithi161.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout:Installing weak dependencies: 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction Summary 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout:============================================================================================ 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout:Install 150 Packages 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout:Upgrade 24 Packages 2024-03-23T14:32:30.206 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:32:30.207 INFO:teuthology.orchestra.run.smithi161.stdout:Total download size: 224 M 2024-03-23T14:32:30.207 INFO:teuthology.orchestra.run.smithi161.stdout:Downloading Packages: 2024-03-23T14:32:30.692 INFO:teuthology.orchestra.run.smithi161.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-03-23T14:32:30.909 INFO:teuthology.orchestra.run.smithi161.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 4.0 MB/s | 888 kB 00:00 2024-03-23T14:32:30.968 INFO:teuthology.orchestra.run.smithi161.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 2.4 MB/s | 145 kB 00:00 2024-03-23T14:32:31.226 INFO:teuthology.orchestra.run.smithi161.stdout:(4/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 9.1 MB/s | 2.4 MB 00:00 2024-03-23T14:32:31.410 INFO:teuthology.orchestra.run.smithi161.stdout:(5/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 8.1 MB/s | 1.5 MB 00:00 2024-03-23T14:32:31.650 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T14:32:31.651 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-23T14:32:31.663 INFO:teuthology.orchestra.run.smithi183.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-23T14:32:31.663 INFO:teuthology.orchestra.run.smithi183.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-23T14:32:31.664 INFO:teuthology.orchestra.run.smithi183.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-23T14:32:31.665 INFO:teuthology.orchestra.run.smithi183.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-23T14:32:31.667 INFO:teuthology.orchestra.run.smithi183.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-23T14:32:31.670 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-23T14:32:31.670 INFO:teuthology.orchestra.run.smithi183.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-23T14:32:31.750 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-23T14:32:31.769 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-23T14:32:31.769 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-23T14:32:31.769 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-23T14:32:31.769 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-23T14:32:31.769 INFO:teuthology.orchestra.run.smithi183.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 6.6 k 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.3 M 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 888 k 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 145 k 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 1.5 M 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 138 k 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 1.8 M 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 7.4 M 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 49 k 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 9.8 M 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 47 M 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 264 k 2024-03-23T14:32:31.770 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 748 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 219 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 27 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 18 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 32 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 744 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 127 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-23T14:32:31.771 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 161 k 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 320 k 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 298 k 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 99 k 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 85 k 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 170 k 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-23T14:32:31.772 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.4 M 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 3.1 M 2024-03-23T14:32:31.773 INFO:teuthology.orchestra.run.smithi183.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-23T14:32:31.774 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 21 M 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 31 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 2.4 M 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 248 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 4.7 M 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 17 M 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1855.g21b3bd67.el9 ceph-noarch 15 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 25 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-23T14:32:31.775 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 163 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 499 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 5.1 M 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-23T14:32:31.776 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-23T14:32:31.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 45 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-1855.g21b3bd67.el9 ceph 131 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-23T14:32:31.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-23T14:32:31.779 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-23T14:32:31.780 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-23T14:32:31.781 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-23T14:32:31.782 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-23T14:32:31.783 INFO:teuthology.orchestra.run.smithi183.stdout:Install 150 Packages 2024-03-23T14:32:31.783 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 24 Packages 2024-03-23T14:32:31.783 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:32:31.783 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 224 M 2024-03-23T14:32:31.783 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-23T14:32:31.960 INFO:teuthology.orchestra.run.smithi161.stdout:(6/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 8.5 MB/s | 4.7 MB 00:00 2024-03-23T14:32:32.186 INFO:teuthology.orchestra.run.smithi161.stdout:(7/174): ceph-common-19.0.0-1855.g21b3bd67.el9. 12 MB/s | 21 MB 00:01 2024-03-23T14:32:32.499 INFO:teuthology.orchestra.run.smithi183.stdout:(1/174): ceph-19.0.0-1855.g21b3bd67.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-03-23T14:32:33.027 INFO:teuthology.orchestra.run.smithi161.stdout:(8/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el9 12 MB/s | 9.8 MB 00:00 2024-03-23T14:32:33.094 INFO:teuthology.orchestra.run.smithi161.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 378 kB/s | 25 kB 00:00 2024-03-23T14:32:33.341 INFO:teuthology.orchestra.run.smithi183.stdout:(2/174): ceph-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.0 MB/s | 888 kB 00:00 2024-03-23T14:32:33.449 INFO:teuthology.orchestra.run.smithi183.stdout:(3/174): ceph-immutable-object-cache-19.0.0-185 1.3 MB/s | 145 kB 00:00 2024-03-23T14:32:33.478 INFO:teuthology.orchestra.run.smithi161.stdout:(10/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x8 11 MB/s | 17 MB 00:01 2024-03-23T14:32:33.578 INFO:teuthology.orchestra.run.smithi161.stdout:(11/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x 1.8 MB/s | 5.3 MB 00:03 2024-03-23T14:32:33.603 INFO:teuthology.orchestra.run.smithi161.stdout:(12/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 1.7 MB/s | 219 kB 00:00 2024-03-23T14:32:33.645 INFO:teuthology.orchestra.run.smithi161.stdout:(13/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 405 kB/s | 27 kB 00:00 2024-03-23T14:32:33.662 INFO:teuthology.orchestra.run.smithi161.stdout:(14/174): libcephfs-devel-19.0.0-1855.g21b3bd67 540 kB/s | 32 kB 00:00 2024-03-23T14:32:33.716 INFO:teuthology.orchestra.run.smithi183.stdout:(4/174): ceph-base-19.0.0-1855.g21b3bd67.el9.x8 3.4 MB/s | 5.3 MB 00:01 2024-03-23T14:32:33.720 INFO:teuthology.orchestra.run.smithi161.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 2.7 MB/s | 163 kB 00:00 2024-03-23T14:32:33.796 INFO:teuthology.orchestra.run.smithi161.stdout:(16/174): librados-devel-19.0.0-1855.g21b3bd67. 1.7 MB/s | 127 kB 00:00 2024-03-23T14:32:33.829 INFO:teuthology.orchestra.run.smithi161.stdout:(17/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 3.9 MB/s | 744 kB 00:00 2024-03-23T14:32:33.873 INFO:teuthology.orchestra.run.smithi161.stdout:(18/174): libradosstriper1-19.0.0-1855.g21b3bd6 6.5 MB/s | 499 kB 00:00 2024-03-23T14:32:33.930 INFO:teuthology.orchestra.run.smithi161.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 774 kB/s | 45 kB 00:00 2024-03-23T14:32:34.005 INFO:teuthology.orchestra.run.smithi161.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.7 MB/s | 131 kB 00:00 2024-03-23T14:32:34.024 INFO:teuthology.orchestra.run.smithi183.stdout:(5/174): ceph-mgr-19.0.0-1855.g21b3bd67.el9.x86 4.8 MB/s | 1.5 MB 00:00 2024-03-23T14:32:34.064 INFO:teuthology.orchestra.run.smithi161.stdout:(21/174): python3-cephfs-19.0.0-1855.g21b3bd67. 2.7 MB/s | 161 kB 00:00 2024-03-23T14:32:34.148 INFO:teuthology.orchestra.run.smithi161.stdout:(22/174): python3-rados-19.0.0-1855.g21b3bd67.e 3.7 MB/s | 320 kB 00:00 2024-03-23T14:32:34.231 INFO:teuthology.orchestra.run.smithi161.stdout:(23/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 3.5 MB/s | 298 kB 00:00 2024-03-23T14:32:34.250 INFO:teuthology.orchestra.run.smithi183.stdout:(6/174): ceph-mds-19.0.0-1855.g21b3bd67.el9.x86 2.9 MB/s | 2.4 MB 00:00 2024-03-23T14:32:34.298 INFO:teuthology.orchestra.run.smithi161.stdout:(24/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.5 MB/s | 99 kB 00:00 2024-03-23T14:32:34.373 INFO:teuthology.orchestra.run.smithi161.stdout:(25/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-23T14:32:34.890 INFO:teuthology.orchestra.run.smithi161.stdout:(26/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 4.8 MB/s | 5.1 MB 00:01 2024-03-23T14:32:34.940 INFO:teuthology.orchestra.run.smithi161.stdout:(27/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 5.4 MB/s | 3.1 MB 00:00 2024-03-23T14:32:34.974 INFO:teuthology.orchestra.run.smithi161.stdout:(28/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 2.0 MB/s | 170 kB 00:00 2024-03-23T14:32:34.999 INFO:teuthology.orchestra.run.smithi161.stdout:(29/174): ceph-grafana-dashboards-19.0.0-1855.g 535 kB/s | 31 kB 00:00 2024-03-23T14:32:35.033 INFO:teuthology.orchestra.run.smithi183.stdout:(7/174): ceph-mon-19.0.0-1855.g21b3bd67.el9.x86 4.7 MB/s | 4.7 MB 00:01 2024-03-23T14:32:35.058 INFO:teuthology.orchestra.run.smithi161.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.6 MB/s | 138 kB 00:00 2024-03-23T14:32:35.366 INFO:teuthology.orchestra.run.smithi161.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 4.9 MB/s | 1.8 MB 00:00 2024-03-23T14:32:35.450 INFO:teuthology.orchestra.run.smithi161.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.9 MB/s | 248 kB 00:00 2024-03-23T14:32:35.525 INFO:teuthology.orchestra.run.smithi161.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 657 kB/s | 49 kB 00:00 2024-03-23T14:32:35.601 INFO:teuthology.orchestra.run.smithi161.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 203 kB/s | 15 kB 00:00 2024-03-23T14:32:35.684 INFO:teuthology.orchestra.run.smithi161.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 3.1 MB/s | 264 kB 00:00 2024-03-23T14:32:35.901 INFO:teuthology.orchestra.run.smithi161.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 3.4 MB/s | 748 kB 00:00 2024-03-23T14:32:35.976 INFO:teuthology.orchestra.run.smithi161.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 242 kB/s | 18 kB 00:00 2024-03-23T14:32:36.168 INFO:teuthology.orchestra.run.smithi161.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.4 MB/s | 662 kB 00:00 2024-03-23T14:32:36.244 INFO:teuthology.orchestra.run.smithi161.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 920 kB 00:00 2024-03-23T14:32:36.277 INFO:teuthology.orchestra.run.smithi161.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 3.1 MB/s | 106 kB 00:00 2024-03-23T14:32:36.319 INFO:teuthology.orchestra.run.smithi161.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 7.2 MB/s | 307 kB 00:00 2024-03-23T14:32:36.344 INFO:teuthology.orchestra.run.smithi161.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 1.2 MB/s | 32 kB 00:00 2024-03-23T14:32:36.453 INFO:teuthology.orchestra.run.smithi161.stdout:(43/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 28 MB/s | 3.0 MB 00:00 2024-03-23T14:32:36.478 INFO:teuthology.orchestra.run.smithi161.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 689 kB/s | 17 kB 00:00 2024-03-23T14:32:36.503 INFO:teuthology.orchestra.run.smithi161.stdout:(45/174): libaio-devel-0.3.111-13.el9.x86_64.rp 405 kB/s | 10 kB 00:00 2024-03-23T14:32:36.537 INFO:teuthology.orchestra.run.smithi161.stdout:(46/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-23T14:32:36.562 INFO:teuthology.orchestra.run.smithi161.stdout:(47/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-23T14:32:36.604 INFO:teuthology.orchestra.run.smithi161.stdout:(48/174): librdkafka-1.6.1-102.el9.x86_64.rpm 15 MB/s | 661 kB 00:00 2024-03-23T14:32:36.642 INFO:teuthology.orchestra.run.smithi183.stdout:(8/174): ceph-osd-19.0.0-1855.g21b3bd67.el9.x86 7.0 MB/s | 17 MB 00:02 2024-03-23T14:32:36.646 INFO:teuthology.orchestra.run.smithi161.stdout:(49/174): libtool-2.4.6-45.el9.x86_64.rpm 14 MB/s | 578 kB 00:00 2024-03-23T14:32:36.671 INFO:teuthology.orchestra.run.smithi161.stdout:(50/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.8 MB/s | 46 kB 00:00 2024-03-23T14:32:36.700 INFO:teuthology.orchestra.run.smithi183.stdout:(9/174): ceph-selinux-19.0.0-1855.g21b3bd67.el9 435 kB/s | 25 kB 00:00 2024-03-23T14:32:36.713 INFO:teuthology.orchestra.run.smithi161.stdout:(51/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-23T14:32:36.747 INFO:teuthology.orchestra.run.smithi161.stdout:(52/174): lua-5.4.4-4.el9.x86_64.rpm 5.4 MB/s | 187 kB 00:00 2024-03-23T14:32:36.789 INFO:teuthology.orchestra.run.smithi161.stdout:(53/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 5.1 MB/s | 217 kB 00:00 2024-03-23T14:32:36.822 INFO:teuthology.orchestra.run.smithi161.stdout:(54/174): openblas-0.3.21-2.el9.x86_64.rpm 1.0 MB/s | 34 kB 00:00 2024-03-23T14:32:36.956 INFO:teuthology.orchestra.run.smithi161.stdout:(55/174): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-03-23T14:32:37.023 INFO:teuthology.orchestra.run.smithi161.stdout:(56/174): postgresql-13.11-1.el9.x86_64.rpm 23 MB/s | 1.5 MB 00:00 2024-03-23T14:32:37.082 INFO:teuthology.orchestra.run.smithi161.stdout:(57/174): postgresql-contrib-13.11-1.el9.x86_64 14 MB/s | 811 kB 00:00 2024-03-23T14:32:37.115 INFO:teuthology.orchestra.run.smithi161.stdout:(58/174): postgresql-private-libs-13.11-1.el9.x 3.9 MB/s | 134 kB 00:00 2024-03-23T14:32:37.207 INFO:teuthology.orchestra.run.smithi161.stdout:(59/174): ceph-mgr-diskprediction-local-19.0.0- 3.4 MB/s | 7.4 MB 00:02 2024-03-23T14:32:37.308 INFO:teuthology.orchestra.run.smithi161.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-23T14:32:37.341 INFO:teuthology.orchestra.run.smithi161.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.7 MB/s | 1.0 MB 00:00 2024-03-23T14:32:37.358 INFO:teuthology.orchestra.run.smithi161.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.6 MB/s | 78 kB 00:00 2024-03-23T14:32:37.433 INFO:teuthology.orchestra.run.smithi183.stdout:(10/174): ceph-radosgw-19.0.0-1855.g21b3bd67.el 4.1 MB/s | 9.8 MB 00:02 2024-03-23T14:32:37.509 INFO:teuthology.orchestra.run.smithi161.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 35 MB/s | 5.8 MB 00:00 2024-03-23T14:32:37.525 INFO:teuthology.orchestra.run.smithi183.stdout:(11/174): cephfs-mirror-19.0.0-1855.g21b3bd67.e 2.3 MB/s | 219 kB 00:00 2024-03-23T14:32:37.534 INFO:teuthology.orchestra.run.smithi161.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 139 kB/s | 24 kB 00:00 2024-03-23T14:32:37.559 INFO:teuthology.orchestra.run.smithi161.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-23T14:32:37.584 INFO:teuthology.orchestra.run.smithi161.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-23T14:32:37.600 INFO:teuthology.orchestra.run.smithi183.stdout:(12/174): cephfs-shell-19.0.0-1855.g21b3bd67.el 361 kB/s | 27 kB 00:00 2024-03-23T14:32:37.610 INFO:teuthology.orchestra.run.smithi161.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 630 kB/s | 32 kB 00:00 2024-03-23T14:32:37.684 INFO:teuthology.orchestra.run.smithi183.stdout:(13/174): libcephfs-devel-19.0.0-1855.g21b3bd67 382 kB/s | 32 kB 00:00 2024-03-23T14:32:37.851 INFO:teuthology.orchestra.run.smithi183.stdout:(14/174): libcephfs2-19.0.0-1855.g21b3bd67.el9. 4.4 MB/s | 744 kB 00:00 2024-03-23T14:32:37.943 INFO:teuthology.orchestra.run.smithi183.stdout:(15/174): libcephsqlite-19.0.0-1855.g21b3bd67.e 1.7 MB/s | 163 kB 00:00 2024-03-23T14:32:38.077 INFO:teuthology.orchestra.run.smithi183.stdout:(16/174): librados-devel-19.0.0-1855.g21b3bd67. 952 kB/s | 127 kB 00:00 2024-03-23T14:32:38.077 INFO:teuthology.orchestra.run.smithi161.stdout:(68/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 9.5 MB/s | 47 MB 00:04 2024-03-23T14:32:38.135 INFO:teuthology.orchestra.run.smithi161.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 8.7 MB/s | 4.8 MB 00:00 2024-03-23T14:32:38.160 INFO:teuthology.orchestra.run.smithi161.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 396 kB/s | 217 kB 00:00 2024-03-23T14:32:38.185 INFO:teuthology.orchestra.run.smithi161.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 174 kB 00:00 2024-03-23T14:32:38.236 INFO:teuthology.orchestra.run.smithi161.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 920 kB/s | 69 kB 00:00 2024-03-23T14:32:38.253 INFO:teuthology.orchestra.run.smithi161.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 450 kB/s | 41 kB 00:00 2024-03-23T14:32:38.277 INFO:teuthology.orchestra.run.smithi183.stdout:(17/174): libradosstriper1-19.0.0-1855.g21b3bd6 2.4 MB/s | 499 kB 00:00 2024-03-23T14:32:38.278 INFO:teuthology.orchestra.run.smithi161.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 2.5 MB/s | 238 kB 00:00 2024-03-23T14:32:38.303 INFO:teuthology.orchestra.run.smithi161.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-23T14:32:38.328 INFO:teuthology.orchestra.run.smithi161.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-23T14:32:38.345 INFO:teuthology.orchestra.run.smithi161.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 710 kB/s | 47 kB 00:00 2024-03-23T14:32:38.370 INFO:teuthology.orchestra.run.smithi161.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 641 kB/s | 43 kB 00:00 2024-03-23T14:32:38.604 INFO:teuthology.orchestra.run.smithi161.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 168 kB/s | 39 kB 00:00 2024-03-23T14:32:38.629 INFO:teuthology.orchestra.run.smithi161.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 160 kB/s | 41 kB 00:00 2024-03-23T14:32:38.821 INFO:teuthology.orchestra.run.smithi161.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-03-23T14:32:38.846 INFO:teuthology.orchestra.run.smithi161.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-03-23T14:32:38.872 INFO:teuthology.orchestra.run.smithi161.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 235 kB/s | 56 kB 00:00 2024-03-23T14:32:38.915 INFO:teuthology.orchestra.run.smithi161.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 672 kB/s | 63 kB 00:00 2024-03-23T14:32:38.939 INFO:teuthology.orchestra.run.smithi161.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2024-03-23T14:32:38.964 INFO:teuthology.orchestra.run.smithi161.stdout:(86/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 479 kB/s | 44 kB 00:00 2024-03-23T14:32:38.989 INFO:teuthology.orchestra.run.smithi161.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.2 MB/s | 241 kB 00:00 2024-03-23T14:32:39.040 INFO:teuthology.orchestra.run.smithi161.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.1 MB 00:00 2024-03-23T14:32:39.065 INFO:teuthology.orchestra.run.smithi161.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 1.7 MB/s | 124 kB 00:00 2024-03-23T14:32:39.090 INFO:teuthology.orchestra.run.smithi161.stdout:(90/174): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-23T14:32:39.115 INFO:teuthology.orchestra.run.smithi161.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-23T14:32:39.252 INFO:teuthology.orchestra.run.smithi183.stdout:(18/174): librgw2-19.0.0-1855.g21b3bd67.el9.x86 5.3 MB/s | 5.1 MB 00:00 2024-03-23T14:32:39.327 INFO:teuthology.orchestra.run.smithi183.stdout:(19/174): python3-ceph-argparse-19.0.0-1855.g21 604 kB/s | 45 kB 00:00 2024-03-23T14:32:39.402 INFO:teuthology.orchestra.run.smithi183.stdout:(20/174): python3-ceph-common-19.0.0-1855.g21b3 1.7 MB/s | 131 kB 00:00 2024-03-23T14:32:39.819 INFO:teuthology.orchestra.run.smithi183.stdout:(21/174): ceph-test-19.0.0-1855.g21b3bd67.el9.x 15 MB/s | 47 MB 00:03 2024-03-23T14:32:39.836 INFO:teuthology.orchestra.run.smithi183.stdout:(22/174): python3-cephfs-19.0.0-1855.g21b3bd67. 372 kB/s | 161 kB 00:00 2024-03-23T14:32:39.887 INFO:teuthology.orchestra.run.smithi183.stdout:(23/174): python3-rados-19.0.0-1855.g21b3bd67.e 4.7 MB/s | 320 kB 00:00 2024-03-23T14:32:39.924 INFO:teuthology.orchestra.run.smithi161.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 218 kB/s | 187 kB 00:00 2024-03-23T14:32:39.945 INFO:teuthology.orchestra.run.smithi183.stdout:(24/174): python3-rbd-19.0.0-1855.g21b3bd67.el9 2.7 MB/s | 298 kB 00:00 2024-03-23T14:32:39.970 INFO:teuthology.orchestra.run.smithi183.stdout:(25/174): python3-rgw-19.0.0-1855.g21b3bd67.el9 1.2 MB/s | 99 kB 00:00 2024-03-23T14:32:40.004 INFO:teuthology.orchestra.run.smithi183.stdout:(26/174): rbd-fuse-19.0.0-1855.g21b3bd67.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-23T14:32:40.007 INFO:teuthology.orchestra.run.smithi161.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 219 kB/s | 195 kB 00:00 2024-03-23T14:32:40.063 INFO:teuthology.orchestra.run.smithi183.stdout:(27/174): rbd-nbd-19.0.0-1855.g21b3bd67.el9.x86 2.9 MB/s | 170 kB 00:00 2024-03-23T14:32:40.107 INFO:teuthology.orchestra.run.smithi161.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 216 kB/s | 21 kB 00:00 2024-03-23T14:32:40.138 INFO:teuthology.orchestra.run.smithi183.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1855.g 419 kB/s | 31 kB 00:00 2024-03-23T14:32:40.158 INFO:teuthology.orchestra.run.smithi161.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 510 kB/s | 25 kB 00:00 2024-03-23T14:32:40.205 INFO:teuthology.orchestra.run.smithi183.stdout:(29/174): rbd-mirror-19.0.0-1855.g21b3bd67.el9. 13 MB/s | 3.1 MB 00:00 2024-03-23T14:32:40.230 INFO:teuthology.orchestra.run.smithi183.stdout:(30/174): ceph-mgr-cephadm-19.0.0-1855.g21b3bd6 1.5 MB/s | 138 kB 00:00 2024-03-23T14:32:40.347 INFO:teuthology.orchestra.run.smithi183.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1855.g21b3b 13 MB/s | 1.8 MB 00:00 2024-03-23T14:32:40.448 INFO:teuthology.orchestra.run.smithi183.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1855.g21 2.4 MB/s | 248 kB 00:00 2024-03-23T14:32:40.523 INFO:teuthology.orchestra.run.smithi183.stdout:(33/174): ceph-mgr-rook-19.0.0-1855.g21b3bd67.e 658 kB/s | 49 kB 00:00 2024-03-23T14:32:40.533 INFO:teuthology.orchestra.run.smithi161.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 11 MB/s | 4.1 MB 00:00 2024-03-23T14:32:40.583 INFO:teuthology.orchestra.run.smithi161.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 4.8 MB/s | 245 kB 00:00 2024-03-23T14:32:40.590 INFO:teuthology.orchestra.run.smithi183.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1855.g2 230 kB/s | 15 kB 00:00 2024-03-23T14:32:40.642 INFO:teuthology.orchestra.run.smithi161.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 4.1 MB/s | 248 kB 00:00 2024-03-23T14:32:40.682 INFO:teuthology.orchestra.run.smithi183.stdout:(35/174): ceph-volume-19.0.0-1855.g21b3bd67.el9 2.8 MB/s | 264 kB 00:00 2024-03-23T14:32:40.692 INFO:teuthology.orchestra.run.smithi161.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-23T14:32:40.718 INFO:teuthology.orchestra.run.smithi161.stdout:(100/174): python3-requests-2.25.1-8.el9.noarch 157 kB/s | 125 kB 00:00 2024-03-23T14:32:40.734 INFO:teuthology.orchestra.run.smithi161.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 832 kB/s | 35 kB 00:00 2024-03-23T14:32:40.785 INFO:teuthology.orchestra.run.smithi161.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-23T14:32:40.790 INFO:teuthology.orchestra.run.smithi183.stdout:(36/174): cephadm-19.0.0-1855.g21b3bd67.el9.noa 6.8 MB/s | 748 kB 00:00 2024-03-23T14:32:40.818 INFO:teuthology.orchestra.run.smithi161.stdout:(103/174): libgfortran-11.4.1-3.el9.x86_64.rpm 472 kB/s | 803 kB 00:01 2024-03-23T14:32:40.835 INFO:teuthology.orchestra.run.smithi161.stdout:(104/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 381 kB/s | 44 kB 00:00 2024-03-23T14:32:40.857 INFO:teuthology.orchestra.run.smithi183.stdout:(37/174): cephfs-top-19.0.0-1855.g21b3bd67.el9. 273 kB/s | 18 kB 00:00 2024-03-23T14:32:40.860 INFO:teuthology.orchestra.run.smithi161.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 297 kB/s | 22 kB 00:00 2024-03-23T14:32:40.936 INFO:teuthology.orchestra.run.smithi161.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-03-23T14:32:41.033 INFO:teuthology.orchestra.run.smithi183.stdout:(38/174): automake-1.16.2-8.el9.noarch.rpm 3.7 MB/s | 662 kB 00:00 2024-03-23T14:32:41.133 INFO:teuthology.orchestra.run.smithi183.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 9.0 MB/s | 920 kB 00:00 2024-03-23T14:32:41.174 INFO:teuthology.orchestra.run.smithi183.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 2.5 MB/s | 106 kB 00:00 2024-03-23T14:32:41.233 INFO:teuthology.orchestra.run.smithi183.stdout:(41/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-23T14:32:41.284 INFO:teuthology.orchestra.run.smithi183.stdout:(42/174): flexiblas-3.0.4-8.el9.x86_64.rpm 633 kB/s | 32 kB 00:00 2024-03-23T14:32:41.328 INFO:teuthology.orchestra.run.smithi161.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 57 kB/s | 19 kB 00:00 2024-03-23T14:32:41.375 INFO:teuthology.orchestra.run.smithi183.stdout:(43/174): ceph-mgr-diskprediction-local-19.0.0- 6.5 MB/s | 7.4 MB 00:01 2024-03-23T14:32:41.468 INFO:teuthology.orchestra.run.smithi183.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 16 MB/s | 3.0 MB 00:00 2024-03-23T14:32:41.492 INFO:teuthology.orchestra.run.smithi183.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 150 kB/s | 17 kB 00:00 2024-03-23T14:32:41.520 INFO:teuthology.orchestra.run.smithi161.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 469 kB/s | 309 kB 00:00 2024-03-23T14:32:41.545 INFO:teuthology.orchestra.run.smithi161.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 773 kB/s | 548 kB 00:00 2024-03-23T14:32:41.628 INFO:teuthology.orchestra.run.smithi161.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 239 kB/s | 26 kB 00:00 2024-03-23T14:32:41.654 INFO:teuthology.orchestra.run.smithi161.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-03-23T14:32:41.685 INFO:teuthology.orchestra.run.smithi183.stdout:(46/174): ceph-common-19.0.0-1855.g21b3bd67.el9 2.2 MB/s | 21 MB 00:09 2024-03-23T14:32:41.710 INFO:teuthology.orchestra.run.smithi183.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 46 kB/s | 10 kB 00:00 2024-03-23T14:32:41.737 INFO:teuthology.orchestra.run.smithi161.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 622 kB/s | 67 kB 00:00 2024-03-23T14:32:41.743 INFO:teuthology.orchestra.run.smithi183.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 636 kB/s | 159 kB 00:00 2024-03-23T14:32:41.768 INFO:teuthology.orchestra.run.smithi183.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 533 kB/s | 44 kB 00:00 2024-03-23T14:32:41.770 INFO:teuthology.orchestra.run.smithi161.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-23T14:32:41.802 INFO:teuthology.orchestra.run.smithi183.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-23T14:32:41.835 INFO:teuthology.orchestra.run.smithi183.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2024-03-23T14:32:41.861 INFO:teuthology.orchestra.run.smithi183.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 518 kB/s | 46 kB 00:00 2024-03-23T14:32:41.886 INFO:teuthology.orchestra.run.smithi183.stdout:(53/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.3 MB/s | 282 kB 00:00 2024-03-23T14:32:41.911 INFO:teuthology.orchestra.run.smithi183.stdout:(54/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-23T14:32:41.929 INFO:teuthology.orchestra.run.smithi161.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 3.4 MB/s | 548 kB 00:00 2024-03-23T14:32:41.936 INFO:teuthology.orchestra.run.smithi183.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-23T14:32:41.961 INFO:teuthology.orchestra.run.smithi183.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 458 kB/s | 34 kB 00:00 2024-03-23T14:32:41.979 INFO:teuthology.orchestra.run.smithi161.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 838 kB 00:00 2024-03-23T14:32:42.037 INFO:teuthology.orchestra.run.smithi183.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 15 MB/s | 1.5 MB 00:00 2024-03-23T14:32:42.046 INFO:teuthology.orchestra.run.smithi161.stdout:(116/174): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-03-23T14:32:42.071 INFO:teuthology.orchestra.run.smithi161.stdout:(117/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 306 kB/s | 43 kB 00:00 2024-03-23T14:32:42.097 INFO:teuthology.orchestra.run.smithi161.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 275 kB/s | 32 kB 00:00 2024-03-23T14:32:42.121 INFO:teuthology.orchestra.run.smithi183.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-03-23T14:32:42.147 INFO:teuthology.orchestra.run.smithi161.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-23T14:32:42.154 INFO:teuthology.orchestra.run.smithi183.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.2 MB/s | 811 kB 00:00 2024-03-23T14:32:42.179 INFO:teuthology.orchestra.run.smithi183.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 959 kB/s | 134 kB 00:00 2024-03-23T14:32:42.189 INFO:teuthology.orchestra.run.smithi161.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-23T14:32:42.214 INFO:teuthology.orchestra.run.smithi161.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.0 MB/s | 358 kB 00:00 2024-03-23T14:32:42.248 INFO:teuthology.orchestra.run.smithi161.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-03-23T14:32:42.290 INFO:teuthology.orchestra.run.smithi161.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 351 kB/s | 35 kB 00:00 2024-03-23T14:32:42.296 INFO:teuthology.orchestra.run.smithi183.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 33 MB/s | 5.7 MB 00:00 2024-03-23T14:32:42.321 INFO:teuthology.orchestra.run.smithi183.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 560 kB/s | 78 kB 00:00 2024-03-23T14:32:42.323 INFO:teuthology.orchestra.run.smithi161.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.8 MB/s | 203 kB 00:00 2024-03-23T14:32:42.355 INFO:teuthology.orchestra.run.smithi183.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-23T14:32:42.407 INFO:teuthology.orchestra.run.smithi161.stdout:(125/174): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-03-23T14:32:42.449 INFO:teuthology.orchestra.run.smithi161.stdout:(126/174): python3-grpcio-tools-1.46.7-10.el9.x 911 kB/s | 144 kB 00:00 2024-03-23T14:32:42.472 INFO:teuthology.orchestra.run.smithi183.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-23T14:32:42.474 INFO:teuthology.orchestra.run.smithi161.stdout:(127/174): python3-jaraco-8.2.1-3.el9.noarch.rp 71 kB/s | 11 kB 00:00 2024-03-23T14:32:42.489 INFO:teuthology.orchestra.run.smithi183.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 147 kB/s | 24 kB 00:00 2024-03-23T14:32:42.505 INFO:teuthology.orchestra.run.smithi183.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 123 kB/s | 18 kB 00:00 2024-03-23T14:32:42.508 INFO:teuthology.orchestra.run.smithi161.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 176 kB/s | 18 kB 00:00 2024-03-23T14:32:42.547 INFO:teuthology.orchestra.run.smithi183.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-23T14:32:42.558 INFO:teuthology.orchestra.run.smithi161.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 213 kB/s | 23 kB 00:00 2024-03-23T14:32:42.564 INFO:teuthology.orchestra.run.smithi183.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 545 kB/s | 32 kB 00:00 2024-03-23T14:32:42.584 INFO:teuthology.orchestra.run.smithi161.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 178 kB/s | 19 kB 00:00 2024-03-23T14:32:42.609 INFO:teuthology.orchestra.run.smithi161.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 195 kB/s | 20 kB 00:00 2024-03-23T14:32:42.656 INFO:teuthology.orchestra.run.smithi183.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-23T14:32:42.681 INFO:teuthology.orchestra.run.smithi183.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.9 MB/s | 217 kB 00:00 2024-03-23T14:32:42.684 INFO:teuthology.orchestra.run.smithi161.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-03-23T14:32:42.707 INFO:teuthology.orchestra.run.smithi183.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-23T14:32:42.709 INFO:teuthology.orchestra.run.smithi161.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 785 kB/s | 79 kB 00:00 2024-03-23T14:32:42.732 INFO:teuthology.orchestra.run.smithi183.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-03-23T14:32:42.751 INFO:teuthology.orchestra.run.smithi161.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 5.3 MB/s | 1.0 MB 00:00 2024-03-23T14:32:42.757 INFO:teuthology.orchestra.run.smithi183.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 826 kB/s | 41 kB 00:00 2024-03-23T14:32:42.782 INFO:teuthology.orchestra.run.smithi183.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-23T14:32:42.784 INFO:teuthology.orchestra.run.smithi161.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-23T14:32:42.807 INFO:teuthology.orchestra.run.smithi183.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-23T14:32:42.810 INFO:teuthology.orchestra.run.smithi161.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.7 MB/s | 272 kB 00:00 2024-03-23T14:32:42.833 INFO:teuthology.orchestra.run.smithi183.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-23T14:32:42.852 INFO:teuthology.orchestra.run.smithi161.stdout:(137/174): python3-portend-3.1.0-2.el9.noarch.r 163 kB/s | 16 kB 00:00 2024-03-23T14:32:42.858 INFO:teuthology.orchestra.run.smithi183.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 630 kB/s | 47 kB 00:00 2024-03-23T14:32:42.883 INFO:teuthology.orchestra.run.smithi183.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 573 kB/s | 43 kB 00:00 2024-03-23T14:32:42.885 INFO:teuthology.orchestra.run.smithi161.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 894 kB/s | 90 kB 00:00 2024-03-23T14:32:42.911 INFO:teuthology.orchestra.run.smithi161.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 281 kB/s | 28 kB 00:00 2024-03-23T14:32:42.933 INFO:teuthology.orchestra.run.smithi183.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 517 kB/s | 39 kB 00:00 2024-03-23T14:32:42.952 INFO:teuthology.orchestra.run.smithi161.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 306 kB/s | 31 kB 00:00 2024-03-23T14:32:43.003 INFO:teuthology.orchestra.run.smithi161.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.6 MB/s | 188 kB 00:00 2024-03-23T14:32:43.028 INFO:teuthology.orchestra.run.smithi161.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 504 kB/s | 59 kB 00:00 2024-03-23T14:32:43.053 INFO:teuthology.orchestra.run.smithi161.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-23T14:32:43.100 INFO:teuthology.orchestra.run.smithi183.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 190 kB/s | 41 kB 00:00 2024-03-23T14:32:43.104 INFO:teuthology.orchestra.run.smithi161.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 508 kB/s | 51 kB 00:00 2024-03-23T14:32:43.129 INFO:teuthology.orchestra.run.smithi161.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-23T14:32:43.154 INFO:teuthology.orchestra.run.smithi161.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 892 kB/s | 90 kB 00:00 2024-03-23T14:32:43.221 INFO:teuthology.orchestra.run.smithi161.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 218 kB/s | 20 kB 00:00 2024-03-23T14:32:43.246 INFO:teuthology.orchestra.run.smithi161.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-03-23T14:32:43.267 INFO:teuthology.orchestra.run.smithi183.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 38 MB/s | 16 MB 00:00 2024-03-23T14:32:43.274 INFO:teuthology.orchestra.run.smithi161.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-03-23T14:32:43.292 INFO:teuthology.orchestra.run.smithi183.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 843 kB/s | 300 kB 00:00 2024-03-23T14:32:43.318 INFO:teuthology.orchestra.run.smithi183.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 263 kB/s | 56 kB 00:00 2024-03-23T14:32:43.343 INFO:teuthology.orchestra.run.smithi183.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 841 kB/s | 63 kB 00:00 2024-03-23T14:32:43.368 INFO:teuthology.orchestra.run.smithi183.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 880 kB/s | 44 kB 00:00 2024-03-23T14:32:43.372 INFO:teuthology.orchestra.run.smithi161.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-23T14:32:43.394 INFO:teuthology.orchestra.run.smithi183.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-23T14:32:43.414 INFO:teuthology.orchestra.run.smithi161.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-23T14:32:43.444 INFO:teuthology.orchestra.run.smithi183.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.4 MB/s | 241 kB 00:00 2024-03-23T14:32:43.478 INFO:teuthology.orchestra.run.smithi183.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-23T14:32:43.502 INFO:teuthology.orchestra.run.smithi183.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 953 kB/s | 103 kB 00:00 2024-03-23T14:32:43.528 INFO:teuthology.orchestra.run.smithi183.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-23T14:32:43.553 INFO:teuthology.orchestra.run.smithi183.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-23T14:32:43.655 INFO:teuthology.orchestra.run.smithi161.stdout:(152/174): librados2-19.0.0-1855.g21b3bd67.el9. 8.3 MB/s | 3.4 MB 00:00 2024-03-23T14:32:43.931 INFO:teuthology.orchestra.run.smithi161.stdout:(153/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 4.8 MB/s | 3.1 MB 00:00 2024-03-23T14:32:44.428 INFO:teuthology.orchestra.run.smithi183.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 223 kB/s | 195 kB 00:00 2024-03-23T14:32:44.453 INFO:teuthology.orchestra.run.smithi183.stdout:(93/174): jq-1.6-16.el9.x86_64.rpm 197 kB/s | 187 kB 00:00 2024-03-23T14:32:44.573 INFO:teuthology.orchestra.run.smithi161.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 68 kB/s | 42 kB 00:00 2024-03-23T14:32:44.629 INFO:teuthology.orchestra.run.smithi183.stdout:(94/174): elfutils-libelf-devel-0.190-2.el9.x86 123 kB/s | 21 kB 00:00 2024-03-23T14:32:44.721 INFO:teuthology.orchestra.run.smithi183.stdout:(95/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 277 kB/s | 25 kB 00:00 2024-03-23T14:32:45.079 INFO:teuthology.orchestra.run.smithi183.stdout:(96/174): openssl-devel-3.0.7-27.el9.x86_64.rpm 12 MB/s | 4.1 MB 00:00 2024-03-23T14:32:45.129 INFO:teuthology.orchestra.run.smithi183.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 501 kB/s | 803 kB 00:01 2024-03-23T14:32:45.155 INFO:teuthology.orchestra.run.smithi183.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 3.2 MB/s | 245 kB 00:00 2024-03-23T14:32:45.196 INFO:teuthology.orchestra.run.smithi183.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 3.6 MB/s | 248 kB 00:00 2024-03-23T14:32:45.247 INFO:teuthology.orchestra.run.smithi183.stdout:(100/174): python3-requests-2.25.1-8.el9.noarch 153 kB/s | 125 kB 00:00 2024-03-23T14:32:45.289 INFO:teuthology.orchestra.run.smithi183.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 531 kB/s | 35 kB 00:00 2024-03-23T14:32:45.381 INFO:teuthology.orchestra.run.smithi183.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.4 MB/s | 94 kB 00:00 2024-03-23T14:32:45.406 INFO:teuthology.orchestra.run.smithi183.stdout:(103/174): qatlib-23.11.0-1.el9.x86_64.rpm 863 kB/s | 216 kB 00:00 2024-03-23T14:32:45.431 INFO:teuthology.orchestra.run.smithi183.stdout:(104/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 243 kB/s | 44 kB 00:00 2024-03-23T14:32:45.490 INFO:teuthology.orchestra.run.smithi183.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 205 kB/s | 22 kB 00:00 2024-03-23T14:32:45.506 INFO:teuthology.orchestra.run.smithi161.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 266 kB/s | 248 kB 00:00 2024-03-23T14:32:45.632 INFO:teuthology.orchestra.run.smithi183.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.7 MB/s | 863 kB 00:00 2024-03-23T14:32:46.090 INFO:teuthology.orchestra.run.smithi183.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-03-23T14:32:46.198 INFO:teuthology.orchestra.run.smithi161.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 104 kB/s | 71 kB 00:00 2024-03-23T14:32:46.350 INFO:teuthology.orchestra.run.smithi183.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 360 kB/s | 309 kB 00:00 2024-03-23T14:32:46.457 INFO:teuthology.orchestra.run.smithi183.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 240 kB/s | 26 kB 00:00 2024-03-23T14:32:46.533 INFO:teuthology.orchestra.run.smithi183.stdout:(110/174): abseil-cpp-20211102.0-3.el9.x86_64.r 498 kB/s | 548 kB 00:01 2024-03-23T14:32:46.566 INFO:teuthology.orchestra.run.smithi183.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 448 kB/s | 49 kB 00:00 2024-03-23T14:32:46.633 INFO:teuthology.orchestra.run.smithi183.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 672 kB/s | 67 kB 00:00 2024-03-23T14:32:46.657 INFO:teuthology.orchestra.run.smithi161.stdout:(157/174): NetworkManager-libnm-1.47.2-1.el9.x8 630 kB/s | 1.8 MB 00:02 2024-03-23T14:32:46.807 INFO:teuthology.orchestra.run.smithi161.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 60 kB/s | 36 kB 00:00 2024-03-23T14:32:46.833 INFO:teuthology.orchestra.run.smithi183.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 566 kB/s | 151 kB 00:00 2024-03-23T14:32:46.965 INFO:teuthology.orchestra.run.smithi161.stdout:(159/174): NetworkManager-1.47.2-1.el9.x86_64.r 665 kB/s | 2.3 MB 00:03 2024-03-23T14:32:47.582 INFO:teuthology.orchestra.run.smithi161.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 208 kB/s | 192 kB 00:00 2024-03-23T14:32:47.616 INFO:teuthology.orchestra.run.smithi161.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 35 kB/s | 23 kB 00:00 2024-03-23T14:32:47.658 INFO:teuthology.orchestra.run.smithi183.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 665 kB/s | 548 kB 00:00 2024-03-23T14:32:47.784 INFO:teuthology.orchestra.run.smithi183.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 729 kB/s | 838 kB 00:01 2024-03-23T14:32:47.800 INFO:teuthology.orchestra.run.smithi183.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 306 kB/s | 43 kB 00:00 2024-03-23T14:32:47.850 INFO:teuthology.orchestra.run.smithi161.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 245 kB/s | 256 kB 00:01 2024-03-23T14:32:47.893 INFO:teuthology.orchestra.run.smithi183.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-03-23T14:32:47.909 INFO:teuthology.orchestra.run.smithi183.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-23T14:32:48.185 INFO:teuthology.orchestra.run.smithi183.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 589 kB/s | 172 kB 00:00 2024-03-23T14:32:48.433 INFO:teuthology.orchestra.run.smithi161.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 127 kB/s | 108 kB 00:00 2024-03-23T14:32:48.493 INFO:teuthology.orchestra.run.smithi183.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 614 kB/s | 358 kB 00:00 2024-03-23T14:32:48.558 INFO:teuthology.orchestra.run.smithi161.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 164 kB/s | 155 kB 00:00 2024-03-23T14:32:48.610 INFO:teuthology.orchestra.run.smithi183.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 302 kB/s | 35 kB 00:00 2024-03-23T14:32:48.652 INFO:teuthology.orchestra.run.smithi183.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 649 kB/s | 303 kB 00:00 2024-03-23T14:32:48.742 INFO:teuthology.orchestra.run.smithi161.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 161 kB/s | 136 kB 00:00 2024-03-23T14:32:48.886 INFO:teuthology.orchestra.run.smithi183.stdout:(123/174): python3-google-auth-2.28.2-1.el9.noa 737 kB/s | 203 kB 00:00 2024-03-23T14:32:49.142 INFO:teuthology.orchestra.run.smithi161.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 90 kB/s | 64 kB 00:00 2024-03-23T14:32:49.176 INFO:teuthology.orchestra.run.smithi161.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 46 kB/s | 29 kB 00:00 2024-03-23T14:32:49.194 INFO:teuthology.orchestra.run.smithi183.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 468 kB/s | 144 kB 00:00 2024-03-23T14:32:49.303 INFO:teuthology.orchestra.run.smithi183.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 98 kB/s | 11 kB 00:00 2024-03-23T14:32:49.395 INFO:teuthology.orchestra.run.smithi183.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 193 kB/s | 18 kB 00:00 2024-03-23T14:32:49.495 INFO:teuthology.orchestra.run.smithi183.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-23T14:32:49.596 INFO:teuthology.orchestra.run.smithi183.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-23T14:32:49.696 INFO:teuthology.orchestra.run.smithi183.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 196 kB/s | 20 kB 00:00 2024-03-23T14:32:50.918 INFO:teuthology.orchestra.run.smithi161.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 570 kB/s | 1.2 MB 00:02 2024-03-23T14:32:51.121 INFO:teuthology.orchestra.run.smithi183.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 734 kB/s | 1.0 MB 00:01 2024-03-23T14:32:51.230 INFO:teuthology.orchestra.run.smithi183.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 440 kB/s | 46 kB 00:00 2024-03-23T14:32:51.355 INFO:teuthology.orchestra.run.smithi183.stdout:(132/174): python3-grpcio-1.46.7-10.el9.x86_64. 772 kB/s | 2.0 MB 00:02 2024-03-23T14:32:51.380 INFO:teuthology.orchestra.run.smithi183.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 525 kB/s | 79 kB 00:00 2024-03-23T14:32:51.456 INFO:teuthology.orchestra.run.smithi183.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 576 kB/s | 58 kB 00:00 2024-03-23T14:32:51.547 INFO:teuthology.orchestra.run.smithi183.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 179 kB/s | 16 kB 00:00 2024-03-23T14:32:51.731 INFO:teuthology.orchestra.run.smithi183.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 491 kB/s | 90 kB 00:00 2024-03-23T14:32:51.756 INFO:teuthology.orchestra.run.smithi183.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 725 kB/s | 272 kB 00:00 2024-03-23T14:32:51.831 INFO:teuthology.orchestra.run.smithi183.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-03-23T14:32:51.848 INFO:teuthology.orchestra.run.smithi183.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 334 kB/s | 31 kB 00:00 2024-03-23T14:32:51.974 INFO:teuthology.orchestra.run.smithi183.stdout:(140/174): libarrow-9.0.0-10.el9.x86_64.rpm 769 kB/s | 4.4 MB 00:05 2024-03-23T14:32:51.990 INFO:teuthology.orchestra.run.smithi183.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-03-23T14:32:52.074 INFO:teuthology.orchestra.run.smithi183.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 356 kB/s | 36 kB 00:00 2024-03-23T14:32:52.099 INFO:teuthology.orchestra.run.smithi183.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 704 kB/s | 188 kB 00:00 2024-03-23T14:32:52.109 INFO:teuthology.orchestra.run.smithi161.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 390 kB/s | 465 kB 00:01 2024-03-23T14:32:52.116 INFO:teuthology.orchestra.run.smithi183.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 407 kB/s | 51 kB 00:00 2024-03-23T14:32:52.294 INFO:teuthology.orchestra.run.smithi183.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 467 kB/s | 90 kB 00:00 2024-03-23T14:32:52.400 INFO:teuthology.orchestra.run.smithi183.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 185 kB/s | 20 kB 00:00 2024-03-23T14:32:52.442 INFO:teuthology.orchestra.run.smithi183.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 625 kB/s | 230 kB 00:00 2024-03-23T14:32:52.534 INFO:teuthology.orchestra.run.smithi161.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 651 kB/s | 2.2 MB 00:03 2024-03-23T14:32:52.676 INFO:teuthology.orchestra.run.smithi183.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 696 kB/s | 192 kB 00:00 2024-03-23T14:32:52.676 INFO:teuthology.orchestra.run.smithi161.stdout:(171/174): util-linux-2.37.4-18.el9.x86_64.rpm 668 kB/s | 2.3 MB 00:03 2024-03-23T14:32:52.701 INFO:teuthology.orchestra.run.smithi183.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 731 kB/s | 427 kB 00:00 2024-03-23T14:32:52.735 INFO:teuthology.orchestra.run.smithi161.stdout:(172/174): util-linux-user-2.37.4-18.el9.x86_64 50 kB/s | 31 kB 00:00 2024-03-23T14:32:52.802 INFO:teuthology.orchestra.run.smithi161.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 595 kB/s | 74 kB 00:00 2024-03-23T14:32:53.185 INFO:teuthology.orchestra.run.smithi183.stdout:(150/174): librbd1-19.0.0-1855.g21b3bd67.el9.x8 6.5 MB/s | 3.1 MB 00:00 2024-03-23T14:32:53.226 INFO:teuthology.orchestra.run.smithi183.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-23T14:32:53.277 INFO:teuthology.orchestra.run.smithi183.stdout:(152/174): librados2-19.0.0-1855.g21b3bd67.el9. 5.6 MB/s | 3.4 MB 00:00 2024-03-23T14:32:54.577 INFO:teuthology.orchestra.run.smithi183.stdout:(153/174): thrift-0.15.0-2.el9.x86_64.rpm 759 kB/s | 1.6 MB 00:02 2024-03-23T14:32:54.710 INFO:teuthology.orchestra.run.smithi161.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 505 kB/s | 1.1 MB 00:02 2024-03-23T14:32:54.715 INFO:teuthology.orchestra.run.smithi161.stdout:-------------------------------------------------------------------------------- 2024-03-23T14:32:54.715 INFO:teuthology.orchestra.run.smithi161.stdout:Total 9.1 MB/s | 224 MB 00:24 2024-03-23T14:32:55.186 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction check 2024-03-23T14:32:55.285 INFO:teuthology.orchestra.run.smithi183.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 59 kB/s | 42 kB 00:00 2024-03-23T14:32:55.464 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction check succeeded. 2024-03-23T14:32:55.464 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction test 2024-03-23T14:32:56.210 INFO:teuthology.orchestra.run.smithi183.stdout:(155/174): NetworkManager-tui-1.47.2-1.el9.x86_ 268 kB/s | 248 kB 00:00 2024-03-23T14:32:56.311 INFO:teuthology.orchestra.run.smithi183.stdout:(156/174): NetworkManager-libnm-1.47.2-1.el9.x8 618 kB/s | 1.8 MB 00:03 2024-03-23T14:32:56.919 INFO:teuthology.orchestra.run.smithi183.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 101 kB/s | 71 kB 00:00 2024-03-23T14:32:56.944 INFO:teuthology.orchestra.run.smithi183.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 58 kB/s | 36 kB 00:00 2024-03-23T14:32:56.995 INFO:teuthology.orchestra.run.smithi183.stdout:(159/174): NetworkManager-1.47.2-1.el9.x86_64.r 626 kB/s | 2.3 MB 00:03 2024-03-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi161.stdout:Transaction test succeeded. 2024-03-23T14:32:57.458 INFO:teuthology.orchestra.run.smithi161.stdout:Running transaction 2024-03-23T14:32:57.620 INFO:teuthology.orchestra.run.smithi183.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 37 kB/s | 23 kB 00:00 2024-03-23T14:32:57.762 INFO:teuthology.orchestra.run.smithi183.stdout:(161/174): elfutils-libelf-0.190-2.el9.x86_64.r 229 kB/s | 192 kB 00:00 2024-03-23T14:32:57.895 INFO:teuthology.orchestra.run.smithi183.stdout:(162/174): elfutils-libs-0.190-2.el9.x86_64.rpm 269 kB/s | 256 kB 00:00 2024-03-23T14:32:58.362 INFO:teuthology.orchestra.run.smithi183.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 146 kB/s | 108 kB 00:00 2024-03-23T14:32:58.563 INFO:teuthology.orchestra.run.smithi183.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 194 kB/s | 155 kB 00:00 2024-03-23T14:32:58.654 INFO:teuthology.orchestra.run.smithi183.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 179 kB/s | 136 kB 00:00 2024-03-23T14:32:59.096 INFO:teuthology.orchestra.run.smithi183.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 87 kB/s | 64 kB 00:00 2024-03-23T14:32:59.205 INFO:teuthology.orchestra.run.smithi183.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 45 kB/s | 29 kB 00:00 2024-03-23T14:33:00.972 INFO:teuthology.orchestra.run.smithi183.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 541 kB/s | 1.2 MB 00:02 2024-03-23T14:33:02.297 INFO:teuthology.orchestra.run.smithi183.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 351 kB/s | 465 kB 00:01 2024-03-23T14:33:02.772 INFO:teuthology.orchestra.run.smithi183.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 601 kB/s | 2.2 MB 00:03 2024-03-23T14:33:02.905 INFO:teuthology.orchestra.run.smithi183.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 51 kB/s | 31 kB 00:00 2024-03-23T14:33:03.006 INFO:teuthology.orchestra.run.smithi183.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 615 kB/s | 2.3 MB 00:03 2024-03-23T14:33:03.164 INFO:teuthology.orchestra.run.smithi183.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 288 kB/s | 74 kB 00:00 2024-03-23T14:33:04.562 INFO:teuthology.orchestra.run.smithi161.stdout: Preparing : 1/1 2024-03-23T14:33:04.665 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-23T14:33:04.708 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-23T14:33:04.745 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T14:33:04.755 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T14:33:04.803 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-23T14:33:04.866 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-23T14:33:04.899 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-23T14:33:04.924 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-23T14:33:04.978 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-23T14:33:05.005 INFO:teuthology.orchestra.run.smithi183.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 492 kB/s | 1.1 MB 00:02 2024-03-23T14:33:05.010 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-23T14:33:05.010 INFO:teuthology.orchestra.run.smithi183.stdout:Total 6.7 MB/s | 224 MB 00:33 2024-03-23T14:33:05.303 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-23T14:33:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-23T14:33:05.470 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-23T14:33:05.492 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-23T14:33:05.501 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-23T14:33:05.599 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-23T14:33:05.633 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-23T14:33:05.682 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-23T14:33:05.769 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-23T14:33:05.769 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-23T14:33:05.859 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-23T14:33:05.911 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-23T14:33:05.947 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-23T14:33:06.050 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-23T14:33:06.073 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-23T14:33:06.138 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-23T14:33:06.177 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-23T14:33:06.199 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T14:33:06.215 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T14:33:06.411 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T14:33:06.449 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T14:33:06.496 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-23T14:33:06.689 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-23T14:33:06.823 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T14:33:06.883 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T14:33:06.993 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T14:33:07.077 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T14:33:07.300 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T14:33:07.410 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-23T14:33:07.501 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-23T14:33:07.539 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-23T14:33:07.565 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-23T14:33:07.601 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-23T14:33:07.675 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-23T14:33:07.729 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-23T14:33:07.754 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-23T14:33:07.754 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-23T14:33:07.766 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-23T14:33:07.831 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-23T14:33:07.865 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-23T14:33:07.909 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-23T14:33:07.950 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-23T14:33:07.978 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-23T14:33:08.047 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-23T14:33:08.082 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-23T14:33:08.126 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-23T14:33:08.155 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-23T14:33:08.224 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-23T14:33:08.271 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-23T14:33:08.344 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-23T14:33:08.477 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-23T14:33:08.532 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-23T14:33:09.051 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-23T14:33:09.126 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-23T14:33:09.204 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-23T14:33:09.255 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T14:33:09.264 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T14:33:09.296 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-23T14:33:09.336 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-23T14:33:09.365 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-23T14:33:09.396 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-23T14:33:09.665 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-23T14:33:09.855 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-23T14:33:09.938 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-23T14:33:10.059 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-23T14:33:10.121 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-23T14:33:10.169 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-23T14:33:10.223 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-23T14:33:10.297 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-23T14:33:10.328 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-23T14:33:10.365 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-23T14:33:10.411 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-23T14:33:10.447 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-23T14:33:10.490 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-23T14:33:10.532 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-23T14:33:10.666 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-23T14:33:10.721 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-23T14:33:10.751 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-23T14:33:10.792 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-23T14:33:10.887 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-23T14:33:10.926 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-23T14:33:10.975 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-23T14:33:11.000 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-23T14:33:11.025 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-23T14:33:11.115 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-23T14:33:11.241 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-23T14:33:11.348 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-23T14:33:11.391 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-23T14:33:11.440 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-23T14:33:11.496 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-23T14:33:11.680 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-23T14:33:11.728 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-23T14:33:11.867 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-23T14:33:11.918 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-23T14:33:11.950 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-23T14:33:11.980 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-23T14:33:12.009 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-23T14:33:12.043 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-23T14:33:12.082 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-23T14:33:12.114 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-23T14:33:12.152 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-23T14:33:12.229 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-23T14:33:12.262 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-23T14:33:12.296 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-23T14:33:12.337 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-23T14:33:12.372 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-23T14:33:12.461 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-23T14:33:12.542 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-23T14:33:12.586 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-23T14:33:12.625 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-23T14:33:12.681 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-23T14:33:12.778 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-23T14:33:12.827 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-23T14:33:13.554 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-23T14:33:13.607 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-23T14:33:13.641 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-23T14:33:13.689 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-23T14:33:13.785 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-23T14:33:13.833 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-23T14:33:14.005 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-23T14:33:14.039 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-23T14:33:14.118 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-23T14:33:14.691 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-23T14:33:14.771 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-23T14:33:14.802 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-23T14:33:14.828 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-23T14:33:14.865 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T14:33:14.875 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-23T14:33:14.894 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-23T14:33:14.921 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-23T14:33:14.977 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-23T14:33:15.010 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-23T14:33:15.036 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-23T14:33:15.086 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-23T14:33:15.418 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-23T14:33:15.470 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-23T14:33:15.590 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-23T14:33:15.611 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-23T14:33:15.703 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-23T14:33:15.742 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-23T14:33:15.787 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-23T14:33:15.988 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-23T14:33:16.046 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-23T14:33:16.074 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-23T14:33:16.140 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-23T14:33:16.178 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-23T14:33:16.200 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-23T14:33:16.222 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-23T14:33:16.253 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-23T14:33:16.265 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 17/198 2024-03-23T14:33:16.284 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-23T14:33:16.304 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-23T14:33:16.311 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-23T14:33:16.327 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T14:33:16.342 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-23T14:33:16.535 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T14:33:16.570 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-23T14:33:16.614 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-23T14:33:16.626 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-23T14:33:16.658 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-23T14:33:16.748 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-23T14:33:16.779 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-23T14:33:16.797 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-23T14:33:16.816 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-23T14:33:16.941 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T14:33:17.003 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-23T14:33:17.092 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T14:33:17.137 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T14:33:17.279 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-23T14:33:17.352 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-23T14:33:17.411 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-23T14:33:17.458 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-23T14:33:17.482 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-23T14:33:17.504 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-23T14:33:17.560 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-23T14:33:17.593 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-23T14:33:17.619 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-23T14:33:17.662 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-23T14:33:17.737 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-23T14:33:17.790 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-23T14:33:17.829 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-23T14:33:17.892 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-23T14:33:17.928 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-23T14:33:17.971 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-23T14:33:18.005 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-23T14:33:18.032 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-23T14:33:18.100 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-23T14:33:18.137 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 39/198 2024-03-23T14:33:18.180 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 40/198 2024-03-23T14:33:18.217 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 41/198 2024-03-23T14:33:18.286 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-23T14:33:18.332 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-23T14:33:18.406 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-23T14:33:18.546 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-23T14:33:18.600 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-23T14:33:19.114 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-23T14:33:19.188 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-23T14:33:19.272 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-23T14:33:19.325 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T14:33:19.350 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-23T14:33:19.374 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-23T14:33:19.422 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-23T14:33:19.460 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 53/198 2024-03-23T14:33:19.490 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-23T14:33:19.733 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-23T14:33:19.774 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-23T14:33:19.775 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-23T14:33:19.972 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-23T14:33:20.050 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-23T14:33:20.103 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-23T14:33:20.247 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-23T14:33:20.324 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-23T14:33:20.330 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-23T14:33:20.362 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-23T14:33:20.385 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-23T14:33:20.453 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-23T14:33:20.453 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T14:33:20.453 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T14:33:20.453 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:20.458 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-23T14:33:20.490 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-23T14:33:20.496 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-23T14:33:20.526 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-23T14:33:20.573 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-23T14:33:20.608 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-23T14:33:20.652 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-23T14:33:20.694 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-23T14:33:20.817 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-23T14:33:20.874 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-23T14:33:20.904 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-23T14:33:20.946 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-23T14:33:21.040 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-23T14:33:21.071 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-23T14:33:21.111 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-23T14:33:21.137 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-23T14:33:21.161 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-23T14:33:21.249 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-23T14:33:21.369 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-23T14:33:21.477 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-23T14:33:21.520 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-23T14:33:21.569 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-23T14:33:21.623 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-23T14:33:21.807 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-23T14:33:21.857 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-23T14:33:21.985 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-23T14:33:22.022 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-23T14:33:22.054 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-23T14:33:22.085 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-23T14:33:22.113 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-23T14:33:22.147 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-23T14:33:22.186 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-23T14:33:22.218 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-23T14:33:22.248 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-23T14:33:22.332 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-23T14:33:22.367 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-23T14:33:22.400 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-23T14:33:22.441 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-23T14:33:22.476 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-23T14:33:22.564 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-23T14:33:22.666 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-23T14:33:22.715 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-23T14:33:22.755 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-23T14:33:22.793 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-23T14:33:22.891 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-23T14:33:22.940 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-23T14:33:23.657 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-23T14:33:23.703 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-23T14:33:23.737 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-23T14:33:23.785 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-23T14:33:23.865 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-23T14:33:23.905 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-23T14:33:24.074 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-23T14:33:24.102 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-23T14:33:24.180 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-23T14:33:24.830 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-23T14:33:24.943 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-23T14:33:26.187 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-23T14:33:26.263 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-23T14:33:26.294 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-23T14:33:26.317 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-23T14:33:26.344 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-23T14:33:26.658 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-23T14:33:26.691 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-23T14:33:26.782 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 124/198 2024-03-23T14:33:26.813 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 125/198 2024-03-23T14:33:26.849 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-23T14:33:27.300 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-23T14:33:27.336 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-23T14:33:27.394 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 128/198 2024-03-23T14:33:27.528 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 129/198 2024-03-23T14:33:30.090 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-23T14:33:30.128 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-23T14:33:30.666 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 130/198 2024-03-23T14:33:30.705 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-23T14:33:30.780 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 131/198 2024-03-23T14:33:30.780 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-23T14:33:30.780 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-23T14:33:30.781 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:30.815 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /sys 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /proc 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /mnt 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /var/tmp 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /home 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /root 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout:skipping the directory /tmp 2024-03-23T14:33:31.899 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:32.059 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-23T14:33:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-23T14:33:32.075 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:32.075 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T14:33:32.075 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:33.578 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-23T14:33:33.604 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-23T14:33:33.604 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:33.605 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T14:33:33.605 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T14:33:33.605 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T14:33:33.605 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:33.837 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-23T14:33:33.859 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-23T14:33:33.860 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:33.860 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T14:33:33.860 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T14:33:33.860 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T14:33:33.860 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:34.304 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-23T14:33:34.327 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-23T14:33:34.327 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:34.327 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T14:33:34.327 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T14:33:34.327 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T14:33:34.328 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:34.423 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-23T14:33:34.882 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-23T14:33:34.913 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-23T14:33:34.944 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-23T14:33:35.005 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-23T14:33:35.027 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-23T14:33:35.260 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-23T14:33:35.320 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-23T14:33:36.293 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-23T14:33:36.329 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-23T14:33:36.430 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-23T14:33:36.593 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-23T14:33:36.622 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-23T14:33:36.646 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-23T14:33:36.646 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:36.646 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T14:33:36.646 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T14:33:36.646 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T14:33:36.647 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:36.690 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-23T14:33:36.701 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-23T14:33:36.756 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-23T14:33:37.619 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-23T14:33:37.650 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-23T14:33:37.673 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-23T14:33:37.673 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:37.673 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T14:33:37.674 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T14:33:37.674 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T14:33:37.674 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:37.717 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-23T14:33:37.739 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-23T14:33:37.739 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:37.740 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T14:33:37.740 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:37.785 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-23T14:33:37.807 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-23T14:33:37.808 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:37.808 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T14:33:37.808 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T14:33:37.808 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T14:33:37.808 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:38.110 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-23T14:33:38.133 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-23T14:33:38.133 INFO:teuthology.orchestra.run.smithi161.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:38.133 INFO:teuthology.orchestra.run.smithi161.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T14:33:38.133 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T14:33:38.133 INFO:teuthology.orchestra.run.smithi161.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T14:33:38.134 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:42.585 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 132/198 2024-03-23T14:33:42.585 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-23T14:33:42.586 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-23T14:33:42.586 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-23T14:33:42.586 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-23T14:33:42.586 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-23T14:33:42.586 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-23T14:33:42.586 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-23T14:33:42.586 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:42.615 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-23T14:33:42.656 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-23T14:33:42.686 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-23T14:33:42.713 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-23T14:33:42.744 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-23T14:33:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 133/198 2024-03-23T14:33:42.760 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:42.760 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-23T14:33:42.760 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:42.896 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-23T14:33:43.007 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-23T14:33:43.052 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-23T14:33:43.099 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-23T14:33:43.153 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-23T14:33:43.188 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-23T14:33:43.220 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-23T14:33:43.252 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-23T14:33:43.282 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-23T14:33:43.349 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-23T14:33:43.399 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-23T14:33:43.430 INFO:teuthology.orchestra.run.smithi161.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-23T14:33:44.013 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T14:33:44.053 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T14:33:44.073 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T14:33:44.239 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-23T14:33:44.264 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 134/198 2024-03-23T14:33:44.265 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:44.265 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-23T14:33:44.265 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T14:33:44.265 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-23T14:33:44.265 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:44.489 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-23T14:33:44.512 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 135/198 2024-03-23T14:33:44.512 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:44.512 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-23T14:33:44.512 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T14:33:44.512 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-23T14:33:44.512 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:44.896 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-23T14:33:44.957 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-23T14:33:44.973 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-23T14:33:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 136/198 2024-03-23T14:33:44.979 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:44.979 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-23T14:33:44.979 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T14:33:44.979 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-23T14:33:44.979 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:45.014 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-23T14:33:45.067 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-23T14:33:45.106 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-23T14:33:45.136 INFO:teuthology.orchestra.run.smithi161.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-23T14:33:45.161 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-23T14:33:45.187 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-23T14:33:45.213 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-23T14:33:45.214 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T14:33:45.301 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T14:33:45.339 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-23T14:33:45.339 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T14:33:45.410 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-23T14:33:45.415 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T14:33:45.432 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 138/198 2024-03-23T14:33:45.448 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-23T14:33:45.455 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 139/198 2024-03-23T14:33:45.483 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-23T14:33:45.508 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-23T14:33:45.515 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 140/198 2024-03-23T14:33:45.533 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-23T14:33:45.538 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-23T14:33:45.567 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-23T14:33:45.606 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-23T14:33:45.638 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-23T14:33:45.663 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-23T14:33:45.689 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-23T14:33:45.722 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-23T14:33:45.747 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-23T14:33:45.754 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T14:33:45.754 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T14:33:45.769 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 141/198 2024-03-23T14:33:45.797 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-23T14:33:45.942 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T14:33:45.942 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T14:33:46.016 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T14:33:46.041 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-23T14:33:46.073 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-23T14:33:46.101 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-23T14:33:46.126 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-23T14:33:46.191 INFO:teuthology.orchestra.run.smithi161.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T14:33:46.760 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 142/198 2024-03-23T14:33:46.811 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-23T14:33:46.910 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 143/198 2024-03-23T14:33:47.068 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 144/198 2024-03-23T14:33:47.092 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-23T14:33:47.116 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 145/198 2024-03-23T14:33:47.116 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:47.116 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-23T14:33:47.116 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T14:33:47.116 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-23T14:33:47.116 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:47.159 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-23T14:33:47.170 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 146/198 2024-03-23T14:33:47.241 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 147/198 2024-03-23T14:33:48.090 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-23T14:33:48.128 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-23T14:33:48.151 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 149/198 2024-03-23T14:33:48.151 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:48.151 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-23T14:33:48.151 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T14:33:48.151 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-23T14:33:48.151 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:48.195 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-23T14:33:48.216 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 150/198 2024-03-23T14:33:48.216 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:48.217 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-23T14:33:48.217 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:48.271 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-23T14:33:48.293 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-23T14:33:48.293 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:48.293 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-23T14:33:48.293 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T14:33:48.293 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-23T14:33:48.294 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:48.596 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-23T14:33:48.618 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 152/198 2024-03-23T14:33:48.619 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-23T14:33:48.619 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-23T14:33:48.619 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T14:33:48.619 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-23T14:33:48.619 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-23T14:33:50.911 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-23T14:33:50.912 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-23T14:33:50.913 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-23T14:33:50.914 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-23T14:33:50.915 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-23T14:33:50.916 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-23T14:33:50.917 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-23T14:33:50.918 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-23T14:33:50.919 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-23T14:33:50.920 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-23T14:33:50.921 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-23T14:33:50.922 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-23T14:33:50.923 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-23T14:33:50.924 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-23T14:33:50.925 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-23T14:33:50.926 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-23T14:33:50.926 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-23T14:33:50.926 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-23T14:33:50.926 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-23T14:33:50.926 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-23T14:33:50.926 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-23T14:33:53.068 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-23T14:33:53.109 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 154/198 2024-03-23T14:33:53.147 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 155/198 2024-03-23T14:33:53.183 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-23T14:33:53.373 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 157/198 2024-03-23T14:33:53.491 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-23T14:33:53.550 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 159/198 2024-03-23T14:33:53.600 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-23T14:33:53.656 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-23T14:33:53.700 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-23T14:33:53.732 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 163/198 2024-03-23T14:33:53.779 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-23T14:33:53.812 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-23T14:33:53.866 INFO:teuthology.orchestra.run.smithi161.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-23T14:33:53.866 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:53.866 INFO:teuthology.orchestra.run.smithi161.stdout:Upgraded: 2024-03-23T14:33:53.866 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T14:33:53.866 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T14:33:53.866 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.867 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout:Installed: 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T14:33:53.868 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.869 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T14:33:53.870 INFO:teuthology.orchestra.run.smithi161.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: jq-1.6-16.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T14:33:53.871 INFO:teuthology.orchestra.run.smithi161.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: lua-5.4.4-4.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-23T14:33:53.872 INFO:teuthology.orchestra.run.smithi161.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T14:33:53.873 INFO:teuthology.orchestra.run.smithi161.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T14:33:53.874 INFO:teuthology.orchestra.run.smithi161.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T14:33:53.875 INFO:teuthology.orchestra.run.smithi161.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T14:33:53.876 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T14:33:53.877 INFO:teuthology.orchestra.run.smithi161.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.878 INFO:teuthology.orchestra.run.smithi161.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:33:53.879 INFO:teuthology.orchestra.run.smithi161.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T14:33:53.879 INFO:teuthology.orchestra.run.smithi161.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T14:33:53.879 INFO:teuthology.orchestra.run.smithi161.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T14:33:53.879 INFO:teuthology.orchestra.run.smithi161.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T14:33:53.879 INFO:teuthology.orchestra.run.smithi161.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T14:33:53.879 INFO:teuthology.orchestra.run.smithi161.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T14:33:53.879 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-23T14:33:53.880 INFO:teuthology.orchestra.run.smithi161.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T14:33:53.880 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:33:53.880 INFO:teuthology.orchestra.run.smithi161.stdout:Complete! 2024-03-23T14:33:53.940 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-23T14:33:53.978 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-23T14:33:54.336 DEBUG:teuthology.parallel:result is None 2024-03-23T14:33:54.528 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T14:33:54.568 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T14:33:54.587 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-23T14:33:55.390 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 170/198 2024-03-23T14:33:55.480 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-23T14:33:55.521 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-23T14:33:55.616 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-23T14:33:55.649 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-23T14:33:55.674 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-23T14:33:55.700 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-23T14:33:55.727 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-23T14:33:55.727 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T14:33:55.797 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-23T14:33:55.836 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-23T14:33:55.836 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T14:33:55.962 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-23T14:33:55.998 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-23T14:33:56.030 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-23T14:33:56.055 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-23T14:33:56.080 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-23T14:33:56.114 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-23T14:33:56.153 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-23T14:33:56.185 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-23T14:33:56.210 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-23T14:33:56.236 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-23T14:33:56.261 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-23T14:33:56.286 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-23T14:33:56.293 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T14:33:56.293 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T14:33:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-23T14:33:56.481 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T14:33:56.564 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-23T14:33:56.589 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-23T14:33:56.621 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-23T14:33:56.648 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-23T14:33:56.673 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-23T14:33:56.739 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 1/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 3/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 4/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1855.g21b3b 5/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 6/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 7/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 8/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 9/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 10/198 2024-03-23T14:34:01.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 11/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 12/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 13/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 14/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_ 15/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 16/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 17/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_6 18/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86 19/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 20/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el 21/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9. 22/198 2024-03-23T14:34:01.582 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_6 23/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 24/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 25/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 26/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 27/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 28/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 29/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67. 30/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noa 31/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.n 32/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b 33/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el 34/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 35/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.e 36/198 2024-03-23T14:34:01.583 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 37/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 38/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 39/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-23T14:34:01.584 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-23T14:34:01.585 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-23T14:34:01.586 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-23T14:34:01.587 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-23T14:34:01.588 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-23T14:34:01.589 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-23T14:34:01.590 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-23T14:34:01.591 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 151/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 153/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-23T14:34:01.592 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-23T14:34:01.593 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-23T14:34:01.594 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-23T14:34:01.595 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-23T14:34:01.596 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-23T14:34:04.538 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: acl-2.3.1-4.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-23T14:34:04.539 INFO:teuthology.orchestra.run.smithi183.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-23T14:34:04.540 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: automake-1.16.2-8.el9.noarch 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: bison-3.7.4-5.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.541 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.542 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-shell-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: cephfs-top-2:19.0.0-1855.g21b3bd67.el9.noarch 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: flex-2.6.4-9.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-23T14:34:04.543 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.544 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-23T14:34:04.545 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-13.11-1.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-23T14:34:04.546 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-23T14:34:04.547 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-23T14:34:04.548 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-23T14:34:04.549 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-23T14:34:04.550 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-23T14:34:04.551 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-1855.g21b3bd67.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-23T14:34:04.552 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-23T14:34:04.553 INFO:teuthology.orchestra.run.smithi183.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-23T14:34:04.553 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:04.553 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-23T14:34:05.022 DEBUG:teuthology.parallel:result is None 2024-03-23T14:34:05.023 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T14:34:05.245 DEBUG:teuthology.orchestra.run.smithi161:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T14:34:05.272 INFO:teuthology.orchestra.run.smithi161.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-23T14:34:05.272 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-23T14:34:05.273 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-23T14:34:05.275 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=21b3bd672163092b4e542f772797fcba02d3d4de 2024-03-23T14:34:05.493 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-23T14:34:05.514 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-1855.g21b3bd67.el9 2024-03-23T14:34:05.515 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1855.g21b3bd67.el9 2024-03-23T14:34:05.515 INFO:teuthology.task.install:The correct ceph version 19.0.0-1855.g21b3bd67 is installed. 2024-03-23T14:34:05.517 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-23T14:34:05.517 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:05.517 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T14:34:05.555 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:05.555 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-23T14:34:05.585 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-23T14:34:05.586 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:05.586 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T14:34:05.625 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T14:34:05.694 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:05.695 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-23T14:34:05.722 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-23T14:34:05.789 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-23T14:34:05.790 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:05.790 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T14:34:05.819 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T14:34:05.888 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:05.888 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-23T14:34:05.915 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-23T14:34:05.981 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-23T14:34:05.982 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:05.982 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T14:34:06.010 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T14:34:06.078 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:06.079 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-23T14:34:06.105 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-23T14:34:06.172 INFO:teuthology.run_tasks:Running task ceph... 2024-03-23T14:34:06.260 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-23T14:34:06.260 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 777 /var/log/ceph 2024-03-23T14:34:06.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2024-03-23T14:34:06.289 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-23T14:34:06.289 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T14:34:06.331 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-23T14:34:06.357 INFO:tasks.ceph:Creating extra log directories... 2024-03-23T14:34:06.357 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T14:34:06.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-23T14:34:06.432 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-03-23T14:34:06.432 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-03-23T14:34:06.432 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-03-23T14:34:06.432 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:06.432 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-23T14:34:06.479 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-23T14:34:06.602 DEBUG:teuthology.orchestra.run.smithi161:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-23T14:34:06.630 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:06.630 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-03-23T14:34:06.645 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-03-23T14:34:06.764 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-03-23T14:34:06.798 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-23T14:34:06.798 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-03-23T14:34:06.798 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384', 'branch': 'wip-batrick-testing-20240323.003144-squid', 'description': 'fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7618384', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-batrick-testing-20240323.003144-squid'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'subvolume_version': 1, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-batrick-testing-20240323.003144-squid', 'sha1': 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 85, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'client.2'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'client.3']], 'seed': 4284, 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'sleep_before_teardown': 0, 'subset': '405/512', 'suite': 'fs', 'suite_branch': 'wip-batrick-testing-20240323.003144-squid', 'suite_path': '/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/batrick/ceph.git', 'suite_sha1': 'd39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi161.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKdulghjUx2dBVzfkJOysFyenP9PDwo02ofmLKEojk9EStLnapx7pnDP8GFdWOec+snZKV2BIzwqhOjM9JWDjM=', 'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC7RNbUBA5eX59Y7zf4jpLUGwBV7jKzulhCtUEFQOIZwAED998+oGq2kxB7gy9IynqcatepIhLGMJoYIJxvDj0='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de'}}, {'ceph': {'create_rbd_pool': False, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability'], 'sha1': '21b3bd672163092b4e542f772797fcba02d3d4de', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_pool_perm']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-23_09:56:53', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-23T14:34:06.799 DEBUG:teuthology.orchestra.run.smithi161:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-23T14:34:06.801 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-23T14:34:06.846 DEBUG:teuthology.orchestra.run.smithi161:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T14:34:06.856 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2024-03-23T14:34:06.913 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:06.913 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T14:34:06.937 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T14:34:06.937 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_1 2024-03-23T14:34:06.991 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T14:34:06.992 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T14:34:06.992 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-23T14:34:06.992 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T14:34:06.992 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T14:34:06.992 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-23 14:33:46.535305379 +0000 2024-03-23T14:34:06.992 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-23 14:31:44.318620745 +0000 2024-03-23T14:34:06.992 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-23 14:31:44.318620745 +0000 2024-03-23T14:34:06.992 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-23 14:31:44.318620745 +0000 2024-03-23T14:34:06.992 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T14:34:07.058 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-23T14:34:07.058 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-23T14:34:07.058 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.0002641 s, 1.9 MB/s 2024-03-23T14:34:07.058 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T14:34:07.114 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_2 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-23 14:33:46.536305359 +0000 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-23 14:31:44.666613483 +0000 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-23 14:31:44.666613483 +0000 2024-03-23T14:34:07.169 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-23 14:31:44.666613483 +0000 2024-03-23T14:34:07.170 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T14:34:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-23T14:34:07.235 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-23T14:34:07.236 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000206028 s, 2.5 MB/s 2024-03-23T14:34:07.237 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T14:34:07.292 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_3 2024-03-23T14:34:07.347 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T14:34:07.347 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T14:34:07.347 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-23T14:34:07.347 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T14:34:07.347 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T14:34:07.347 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-23 14:33:46.536305359 +0000 2024-03-23T14:34:07.348 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-23 14:31:45.011606284 +0000 2024-03-23T14:34:07.348 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-23 14:31:45.011606284 +0000 2024-03-23T14:34:07.348 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-23 14:31:45.011606284 +0000 2024-03-23T14:34:07.348 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T14:34:07.413 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-23T14:34:07.414 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-23T14:34:07.414 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.00022402 s, 2.3 MB/s 2024-03-23T14:34:07.415 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T14:34:07.470 DEBUG:teuthology.orchestra.run.smithi161:> stat /dev/vg_nvme/lv_4 2024-03-23T14:34:07.524 INFO:teuthology.orchestra.run.smithi161.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T14:34:07.524 INFO:teuthology.orchestra.run.smithi161.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T14:34:07.524 INFO:teuthology.orchestra.run.smithi161.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-23T14:34:07.524 INFO:teuthology.orchestra.run.smithi161.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T14:34:07.524 INFO:teuthology.orchestra.run.smithi161.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T14:34:07.524 INFO:teuthology.orchestra.run.smithi161.stdout:Access: 2024-03-23 14:33:46.537305338 +0000 2024-03-23T14:34:07.525 INFO:teuthology.orchestra.run.smithi161.stdout:Modify: 2024-03-23 14:31:45.358599042 +0000 2024-03-23T14:34:07.525 INFO:teuthology.orchestra.run.smithi161.stdout:Change: 2024-03-23 14:31:45.358599042 +0000 2024-03-23T14:34:07.525 INFO:teuthology.orchestra.run.smithi161.stdout: Birth: 2024-03-23 14:31:45.358599042 +0000 2024-03-23T14:34:07.525 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T14:34:07.589 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records in 2024-03-23T14:34:07.590 INFO:teuthology.orchestra.run.smithi161.stderr:1+0 records out 2024-03-23T14:34:07.590 INFO:teuthology.orchestra.run.smithi161.stderr:512 bytes copied, 0.000181187 s, 2.8 MB/s 2024-03-23T14:34:07.591 DEBUG:teuthology.orchestra.run.smithi161:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T14:34:07.647 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:07.647 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:07.647 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-23T14:34:07.666 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-23T14:34:07.666 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-23T14:34:07.724 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-23T14:34:07.724 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T14:34:07.724 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-23T14:34:07.724 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T14:34:07.724 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T14:34:07.724 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-23 14:33:57.108541286 +0000 2024-03-23T14:34:07.725 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-23 14:31:43.773022491 +0000 2024-03-23T14:34:07.725 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-23 14:31:43.773022491 +0000 2024-03-23T14:34:07.725 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-23 14:31:43.773022491 +0000 2024-03-23T14:34:07.725 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-23T14:34:07.794 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-23T14:34:07.794 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-23T14:34:07.794 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000204325 s, 2.5 MB/s 2024-03-23T14:34:07.796 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-23T14:34:07.854 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-23T14:34:07.912 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-23T14:34:07.912 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T14:34:07.912 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-23T14:34:07.912 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T14:34:07.912 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T14:34:07.912 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-23 14:33:57.109541259 +0000 2024-03-23T14:34:07.913 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-23 14:31:44.120012957 +0000 2024-03-23T14:34:07.913 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-23 14:31:44.120012957 +0000 2024-03-23T14:34:07.913 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-23 14:31:44.120012957 +0000 2024-03-23T14:34:07.913 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-23T14:34:07.981 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-23T14:34:07.981 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-23T14:34:07.981 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000288707 s, 1.8 MB/s 2024-03-23T14:34:07.982 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-23T14:34:08.039 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-23T14:34:08.095 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-23T14:34:08.096 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T14:34:08.096 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-23T14:34:08.096 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T14:34:08.096 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T14:34:08.096 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-23 14:33:57.109541259 +0000 2024-03-23T14:34:08.096 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-23 14:31:44.452003835 +0000 2024-03-23T14:34:08.096 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-23 14:31:44.452003835 +0000 2024-03-23T14:34:08.096 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-23 14:31:44.452003835 +0000 2024-03-23T14:34:08.096 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-23T14:34:08.162 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-23T14:34:08.163 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-23T14:34:08.163 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000192664 s, 2.7 MB/s 2024-03-23T14:34:08.164 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-23T14:34:08.220 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-23T14:34:08.276 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-23T14:34:08.276 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-23T14:34:08.276 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-23T14:34:08.276 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-23T14:34:08.276 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-23T14:34:08.276 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-23 14:33:57.109541259 +0000 2024-03-23T14:34:08.276 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-23 14:31:44.808994026 +0000 2024-03-23T14:34:08.276 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-23 14:31:44.808994026 +0000 2024-03-23T14:34:08.277 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-23 14:31:44.808994026 +0000 2024-03-23T14:34:08.277 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-23T14:34:08.342 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-23T14:34:08.343 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-23T14:34:08.343 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000208076 s, 2.5 MB/s 2024-03-23T14:34:08.344 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-23T14:34:08.400 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:08.400 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-23T14:34:08.400 INFO:tasks.ceph:Generating config... 2024-03-23T14:34:08.402 INFO:tasks.ceph:[client] client mount timeout = 600 2024-03-23T14:34:08.402 INFO:tasks.ceph:[client] debug client = 20 2024-03-23T14:34:08.402 INFO:tasks.ceph:[client] debug ms = 1 2024-03-23T14:34:08.402 INFO:tasks.ceph:[client] pid file = 2024-03-23T14:34:08.402 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-03-23T14:34:08.402 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-03-23T14:34:08.402 INFO:tasks.ceph:[mds] debug mds = 20 2024-03-23T14:34:08.402 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-03-23T14:34:08.402 INFO:tasks.ceph:[mds] debug ms = 1 2024-03-23T14:34:08.402 INFO:tasks.ceph:[mds] mds debug frag = True 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mgr] debug client = 20 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-03-23T14:34:08.403 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-03-23T14:34:08.403 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-23T14:34:08.404 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-03-23T14:34:08.405 INFO:tasks.ceph:Setting up mon.a... 2024-03-23T14:34:08.405 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-23T14:34:08.466 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.keyring 2024-03-23T14:34:08.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-23T14:34:08.514 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T14:34:08.540 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.161'), ('mon.b', '172.21.15.183'), ('mon.c', '[v2:172.21.15.183:3301,v1:172.21.15.183:6790]')] 2024-03-23T14:34:08.541 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.161,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'pid file': '', 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.b': {}, 'mds.d': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-03-23T14:34:08.542 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:08.542 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T14:34:08.595 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.161 --add b 172.21.15.183 --addv c '[v2:172.21.15.183:3301,v1:172.21.15.183:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:08.680 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:08.680 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: generated fsid 9a16b48f-76af-4198-8fe3-d2c20a776a3e 2024-03-23T14:34:08.680 INFO:teuthology.orchestra.run.smithi161.stdout:setting min_mon_release = quincy 2024-03-23T14:34:08.680 INFO:teuthology.orchestra.run.smithi161.stdout:epoch 0 2024-03-23T14:34:08.680 INFO:teuthology.orchestra.run.smithi161.stdout:fsid 9a16b48f-76af-4198-8fe3-d2c20a776a3e 2024-03-23T14:34:08.681 INFO:teuthology.orchestra.run.smithi161.stdout:last_changed 2024-03-23T14:34:08.685632+0000 2024-03-23T14:34:08.681 INFO:teuthology.orchestra.run.smithi161.stdout:created 2024-03-23T14:34:08.685632+0000 2024-03-23T14:34:08.681 INFO:teuthology.orchestra.run.smithi161.stdout:min_mon_release 17 (quincy) 2024-03-23T14:34:08.681 INFO:teuthology.orchestra.run.smithi161.stdout:election_strategy: 1 2024-03-23T14:34:08.681 INFO:teuthology.orchestra.run.smithi161.stdout:0: [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] mon.a 2024-03-23T14:34:08.681 INFO:teuthology.orchestra.run.smithi161.stdout:1: [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] mon.b 2024-03-23T14:34:08.681 INFO:teuthology.orchestra.run.smithi161.stdout:2: [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] mon.c 2024-03-23T14:34:08.681 INFO:teuthology.orchestra.run.smithi161.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-23T14:34:08.682 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-23T14:34:08.697 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9a16b48f-76af-4198-8fe3-d2c20a776a3e... 2024-03-23T14:34:08.698 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T14:34:08.739 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout:[global] 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: chdir = "" 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: auth supported = cephx 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: filestore xattr use omap = true 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: mon clock drift allowed = 1.000 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: osd crush chooseleaf type = 0 2024-03-23T14:34:08.787 INFO:teuthology.orchestra.run.smithi161.stdout: auth debug = true 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on old message = true 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: ms die on bug = true 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: mon pg warn max object skew = 0 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default size = 2 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.788 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow primary affinity = true 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd allow pg remap = true 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on legacy crush tunables = false 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on crush straw calc version zero = false 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on no sortbitwise = false 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on osd down out interval zero = false 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon warn on too few osds = false 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: mon_allow_pool_size_one = true 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T14:34:08.789 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: osd default data pool replay window = 5 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: mon allow pool delete = true 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: mon cluster log file level = debug 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: debug asserts on shutdown = true 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: mon health detail to clog = false 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: mon host = "172.21.15.161,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: fsid = 9a16b48f-76af-4198-8fe3-d2c20a776a3e 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout:[osd] 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: osd journal size = 100 2024-03-23T14:34:08.790 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub load threshold = 5.0 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: osd scrub max interval = 600 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: osd mclock profile = high_recovery_ops 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: osd recover clone overlap = true 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: osd recovery max chunk = 1048576 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug shutdown = true 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug op order = true 2024-03-23T14:34:08.791 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug verify stray on activate = true 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug trim objects = true 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: osd open classes on start = true 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug pg log writeout = true 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: osd deep scrub update digest min age = 30 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: osd map max advance = 10 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: journal zero on create = true 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: filestore ondisk finisher threads = 3 2024-03-23T14:34:08.792 INFO:teuthology.orchestra.run.smithi161.stdout: filestore apply finisher threads = 3 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: bdev debug aio = true 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: osd debug misdirected ops = true 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: bdev async discard = True 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: bdev enable discard = True 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore allocator = bitmap 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore block size = 96636764160 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: bluestore fsck on mount = True 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluefs = 1/20 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: debug bluestore = 1/20 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: debug osd = 20 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: debug rocksdb = 4/10 2024-03-23T14:34:08.793 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd full ratio = 0.9 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd nearfull ratio = 0.8 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: osd failsafe full ratio = 0.95 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: osd objectstore = bluestore 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: osd op complaint time = 180 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr] 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: debug mgr = 20 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-03-23T14:34:08.794 INFO:teuthology.orchestra.run.smithi161.stdout: mgr/telemetry/nag = false 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: debug client = 20 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout:[mon] 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: debug mon = 20 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: debug paxos = 20 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: debug auth = 20 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: mon data avail warn = 5 2024-03-23T14:34:08.795 INFO:teuthology.orchestra.run.smithi161.stdout: mon mgr mkfs grace = 240 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min pgs per osd = 4 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd reporter subtree level = osd 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: mon osd prime pg temp = true 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: mon reweight min bytes per osd = 10 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T14:34:08.796 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: # 1m isn't quite enough 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: mon_down_mkfs_grace = 2m 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: mon_warn_on_filestore_osds = false 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: mon down mkfs grace = 300 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: mon op complaint time = 120 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout:[client] 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: rgw cache enabled = true 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable ops log = true 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: rgw enable usage log = true 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: client mount timeout = 600 2024-03-23T14:34:08.797 INFO:teuthology.orchestra.run.smithi161.stdout: debug client = 20 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: pid file = "" 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: rados mon op timeout = 900 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: rados osd op timeout = 900 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.a] 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.b] 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout:[mon.c] 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.a] 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.c] 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.b] 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.d] 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout:[mds] 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds = 20 2024-03-23T14:34:08.798 INFO:teuthology.orchestra.run.smithi161.stdout: debug mds balancer = 20 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: debug ms = 1 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: mds debug frag = True 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: mds debug scatterstat = True 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: mds op complaint time = 180 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: mds verify scatter = True 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: osd op complaint time = 180 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: rados mon op timeout = 900 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi161.stdout: rados osd op timeout = 900 2024-03-23T14:34:08.799 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-23T14:34:08.800 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-23T14:34:08.801 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "172.21.15.161,172.21.15.183,[v2:172.21.15.183:3301,v1:172.21.15.183:6790]" 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 9a16b48f-76af-4198-8fe3-d2c20a776a3e 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-23T14:34:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-23T14:34:08.804 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.805 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-23T14:34:08.805 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.805 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-23T14:34:08.805 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.805 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-23T14:34:08.805 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.805 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-23T14:34:08.805 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = bitmap 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1/20 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1/20 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-23T14:34:08.806 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 4/10 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-23T14:34:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-23T14:34:08.808 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-23T14:34:08.809 INFO:teuthology.orchestra.run.smithi183.stdout: mon down mkfs grace = 300 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: mon op complaint time = 120 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: client mount timeout = 600 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: debug client = 20 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = "" 2024-03-23T14:34:08.810 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout:[mds] 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds = 20 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout: debug mds balancer = 20 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug frag = True 2024-03-23T14:34:08.811 INFO:teuthology.orchestra.run.smithi183.stdout: mds debug scatterstat = True 2024-03-23T14:34:08.812 INFO:teuthology.orchestra.run.smithi183.stdout: mds op complaint time = 180 2024-03-23T14:34:08.812 INFO:teuthology.orchestra.run.smithi183.stdout: mds verify scatter = True 2024-03-23T14:34:08.812 INFO:teuthology.orchestra.run.smithi183.stdout: osd op complaint time = 180 2024-03-23T14:34:08.812 INFO:teuthology.orchestra.run.smithi183.stdout: rados mon op timeout = 900 2024-03-23T14:34:08.812 INFO:teuthology.orchestra.run.smithi183.stdout: rados osd op timeout = 900 2024-03-23T14:34:08.813 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-23T14:34:08.814 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-23T14:34:08.869 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-23T14:34:08.869 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:08.869 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-23T14:34:08.884 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:08.884 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-23T14:34:08.939 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:34:08.939 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:08.939 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-23T14:34:08.940 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T14:34:09.018 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:09.019 DEBUG:teuthology.orchestra.run.smithi161:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:09.073 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:34:09.073 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:09.073 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-23T14:34:09.074 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-23T14:34:09.115 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:09.115 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:09.171 INFO:tasks.ceph:Setting up mon nodes... 2024-03-23T14:34:09.171 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-23T14:34:09.171 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-23T14:34:09.226 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-23T14:34:09.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-23T14:34:09.299 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-23T14:34:09.301 INFO:tasks.ceph:Setting up mds nodes... 2024-03-23T14:34:09.301 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-03-23T14:34:09.357 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-03-23T14:34:09.360 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-03-23T14:34:09.387 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-03-23T14:34:09.482 INFO:teuthology.orchestra.run.smithi161.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-03-23T14:34:09.484 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-03-23T14:34:09.511 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-03-23T14:34:09.568 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-03-23T14:34:09.570 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-03-23T14:34:09.597 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-03-23T14:34:09.692 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-03-23T14:34:09.694 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-03-23T14:34:09.721 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-23T14:34:09.721 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-23T14:34:09.763 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-23T14:34:09.779 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-03-23T14:34:09.863 INFO:teuthology.orchestra.run.smithi161.stdout:creating /etc/ceph/ceph.client.2.keyring 2024-03-23T14:34:09.878 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-23T14:34:09.921 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-23T14:34:09.936 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.3 /etc/ceph/ceph.client.3.keyring && sudo chmod 0644 /etc/ceph/ceph.client.3.keyring 2024-03-23T14:34:10.018 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.3.keyring 2024-03-23T14:34:10.033 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-23T14:34:10.034 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi161.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-23T14:34:10.034 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-23T14:34:10.061 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:10.061 INFO:tasks.ceph:role: osd.0 2024-03-23T14:34:10.061 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:34:10.061 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T14:34:10.136 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T14:34:10.642 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-23T14:34:10.644 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-23T14:34:10.644 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-23T14:34:10.797 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-23T14:34:10.834 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-23T14:34:10.904 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:10.904 INFO:tasks.ceph:role: osd.1 2024-03-23T14:34:10.904 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:34:10.904 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T14:34:10.980 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T14:34:11.424 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-23T14:34:11.427 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-23T14:34:11.427 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-23T14:34:11.524 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-23T14:34:11.557 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-23T14:34:11.626 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:11.627 INFO:tasks.ceph:role: osd.2 2024-03-23T14:34:11.627 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:34:11.627 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T14:34:11.702 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T14:34:11.702 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T14:34:11.702 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T14:34:11.702 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T14:34:11.702 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T14:34:11.702 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-23T14:34:11.703 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T14:34:11.703 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T14:34:11.703 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T14:34:11.703 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T14:34:12.218 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-23T14:34:12.221 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-23T14:34:12.221 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-23T14:34:12.349 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-23T14:34:12.383 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-23T14:34:12.452 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:12.452 INFO:tasks.ceph:role: osd.3 2024-03-23T14:34:12.452 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:34:12.452 DEBUG:teuthology.orchestra.run.smithi161:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-23T14:34:12.528 INFO:teuthology.orchestra.run.smithi161.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout: = sunit=0 swidth=0 blks 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T14:34:12.529 INFO:teuthology.orchestra.run.smithi161.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T14:34:13.041 INFO:teuthology.orchestra.run.smithi161.stdout:Discarding blocks...Done. 2024-03-23T14:34:13.043 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi161.front.sepia.ceph.com -o noatime 2024-03-23T14:34:13.044 DEBUG:teuthology.orchestra.run.smithi161:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-23T14:34:13.181 DEBUG:teuthology.orchestra.run.smithi161:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-23T14:34:13.215 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:13.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:13.322+0000 7f8038ec2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-23T14:34:13.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:13.322+0000 7f8038ec2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-23T14:34:13.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:13.322+0000 7f8038ec2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T14:34:13.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:13.322+0000 7f8038ec2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T14:34:13.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:13.322+0000 7f8038ec2640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-23T14:34:13.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:13.322+0000 7f8038ec2640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-23T14:34:16.172 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-23T14:34:16.203 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:16.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:16.299+0000 7f061911b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-23T14:34:16.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:16.299+0000 7f061911b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-23T14:34:16.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:16.299+0000 7f061911b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T14:34:16.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:16.299+0000 7f061911b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T14:34:16.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:16.299+0000 7f061911b640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-23T14:34:16.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:16.300+0000 7f061911b640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-23T14:34:19.137 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-23T14:34:19.168 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:19.265+0000 7fd6d18d9640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-23T14:34:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:19.265+0000 7fd6d18d9640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-23T14:34:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:19.265+0000 7fd6d18d9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T14:34:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:19.265+0000 7fd6d18d9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T14:34:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:19.265+0000 7fd6d18d9640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-23T14:34:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:19.266+0000 7fd6d18d9640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-23T14:34:22.118 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-23T14:34:22.147 DEBUG:teuthology.orchestra.run.smithi161:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:22.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:22.239+0000 7fb35f793640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-23T14:34:22.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:22.240+0000 7fb35f793640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-23T14:34:22.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:22.240+0000 7fb35f793640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T14:34:22.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:22.240+0000 7fb35f793640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T14:34:22.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:22.240+0000 7fb35f793640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-23T14:34:22.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:22.240+0000 7fb35f793640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-23T14:34:25.083 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-23T14:34:25.113 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-23T14:34:25.146 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:25.146 INFO:tasks.ceph:role: osd.4 2024-03-23T14:34:25.146 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:34:25.146 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-23T14:34:25.223 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T14:34:25.224 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T14:34:25.734 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-23T14:34:25.736 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-23T14:34:25.737 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-23T14:34:25.859 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-23T14:34:25.895 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-23T14:34:25.965 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:25.965 INFO:tasks.ceph:role: osd.5 2024-03-23T14:34:25.965 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:34:25.965 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T14:34:26.042 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T14:34:26.043 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T14:34:26.493 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-23T14:34:26.496 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-23T14:34:26.496 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-23T14:34:26.598 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-23T14:34:26.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-23T14:34:26.700 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:26.700 INFO:tasks.ceph:role: osd.6 2024-03-23T14:34:26.700 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:34:26.700 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T14:34:26.776 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T14:34:27.297 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-23T14:34:27.299 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-23T14:34:27.299 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-23T14:34:27.433 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-23T14:34:27.468 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-23T14:34:27.538 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-23T14:34:27.538 INFO:tasks.ceph:role: osd.7 2024-03-23T14:34:27.538 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:34:27.538 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-23T14:34:27.615 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-23T14:34:27.615 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-23T14:34:27.615 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-23T14:34:27.615 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-23T14:34:27.615 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-23T14:34:27.615 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-23T14:34:27.615 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-23T14:34:27.616 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-23T14:34:27.616 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-23T14:34:27.616 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-23T14:34:28.123 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-23T14:34:28.125 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-23T14:34:28.125 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-23T14:34:28.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-23T14:34:28.270 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:28.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:28.380+0000 7fe0364a8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-23T14:34:28.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:28.381+0000 7fe0364a8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-23T14:34:28.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:28.381+0000 7fe0364a8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T14:34:28.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:28.381+0000 7fe0364a8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T14:34:28.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:28.381+0000 7fe0364a8640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-23T14:34:28.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:28.381+0000 7fe0364a8640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-23T14:34:31.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-23T14:34:31.267 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:31.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:31.362+0000 7f4ade906640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-23T14:34:31.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:31.362+0000 7f4ade906640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-23T14:34:31.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:31.362+0000 7f4ade906640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T14:34:31.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:31.362+0000 7f4ade906640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T14:34:31.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:31.362+0000 7f4ade906640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-23T14:34:31.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:31.363+0000 7f4ade906640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-23T14:34:34.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-23T14:34:34.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:34.342 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:34.346+0000 7f6d6c224640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-23T14:34:34.342 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:34.346+0000 7f6d6c224640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-23T14:34:34.342 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:34.346+0000 7f6d6c224640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T14:34:34.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:34.346+0000 7f6d6c224640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T14:34:34.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:34.346+0000 7f6d6c224640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-23T14:34:34.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:34.347+0000 7f6d6c224640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-23T14:34:36.801 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:34:36.804 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:34:37.189 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-23T14:34:37.218 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:37.306 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:37.310+0000 7f479ddfa640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-23T14:34:37.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:37.310+0000 7f479ddfa640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-23T14:34:37.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:37.310+0000 7f479ddfa640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T14:34:37.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:37.311+0000 7f479ddfa640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T14:34:37.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:37.311+0000 7f479ddfa640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-23T14:34:37.307 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:37.311+0000 7f479ddfa640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-23T14:34:40.167 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-23T14:34:40.198 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-23T14:34:40.198 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:40.199 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-23T14:34:40.230 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:40.231 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-03-23T14:34:40.302 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:40.302 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-03-23T14:34:40.372 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:40.372 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-23T14:34:40.441 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:40.441 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-23T14:34:40.510 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:40.510 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-23T14:34:40.579 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:40.579 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-23T14:34:40.647 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:40.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-23T14:34:40.678 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:40.678 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-03-23T14:34:40.748 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:40.748 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-03-23T14:34:40.817 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:40.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-23T14:34:40.886 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:40.886 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-23T14:34:40.954 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:40.954 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-23T14:34:41.022 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:41.022 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-23T14:34:41.089 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:41.089 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-23T14:34:41.105 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:41.106 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/etc/ceph/ceph.client.2.keyring of=/dev/stdout 2024-03-23T14:34:41.161 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:41.161 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-23T14:34:41.176 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:41.176 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.3.keyring of=/dev/stdout 2024-03-23T14:34:41.231 INFO:tasks.ceph:Adding keys to all mons... 2024-03-23T14:34:41.231 DEBUG:teuthology.orchestra.run.smithi161:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-23T14:34:41.233 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-23T14:34:41.274 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.y] 2024-03-23T14:34:41.274 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lMbfQDRAA7IKJtMZ6Z77M6m4dD75Vcg== 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.a] 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lzaKfFRAAc+l/nk//V/gpMXQ1JO3ATg== 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.c] 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lOl8SHRAAPqXulseV9GlBMsoYeIUt5Q== 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.0] 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBl6P5lv7pCExAACgyFS3tWsnorh+23JfVMFQ== 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.1] 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBo6P5lN9XjERAAY7Doq86QfXJ46LU3H9ET4g== 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.2] 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBr6P5le9jdDxAAF7YY5jy4+03N9tB79FqgxA== 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.3] 2024-03-23T14:34:41.275 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBu6P5lfyVdDhAA6GSFuGvPYAUCczsnMpXFkQ== 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout:[mgr.x] 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lz/YnEhAA4Q/N4Xnmj7rkBGQH0WFU5w== 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.b] 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lcMUxIhAAA7sdubbg/sFxR2JSLj6knA== 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout:[mds.d] 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5la0WTKRAACwJ5zIvawF/Gsu3pgRkjRQ== 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.4] 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQB06P5lxMy+FhAAhO1d5E1ioKlC3bqbkingpA== 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.5] 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQB36P5lHl2nFRAA18JrgqJgj6mYykRjhemRvQ== 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.6] 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQB66P5lC5OzFBAAeJslkQ08ZxK2fx9yQKsCOA== 2024-03-23T14:34:41.276 INFO:teuthology.orchestra.run.smithi161.stdout:[osd.7] 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQB96P5lyiCSEhAAn+5LSA94oOhlckTfTUd7Lw== 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout:[client.2] 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lIYbHMxAAOYcn/3VZTDW2K3OBKfjxkA== 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout:[client.1] 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lskU4NxAAed0RZpsIuzbux+yi642hsw== 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout:[client.3] 2024-03-23T14:34:41.277 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBi6P5lwCRzARAAlFyun7fjX8hB/KsVJhLfWg== 2024-03-23T14:34:41.298 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.y] 2024-03-23T14:34:41.298 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5lMbfQDRAA7IKJtMZ6Z77M6m4dD75Vcg== 2024-03-23T14:34:41.299 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.a] 2024-03-23T14:34:41.299 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5lzaKfFRAAc+l/nk//V/gpMXQ1JO3ATg== 2024-03-23T14:34:41.299 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.c] 2024-03-23T14:34:41.299 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5lOl8SHRAAPqXulseV9GlBMsoYeIUt5Q== 2024-03-23T14:34:41.299 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-23T14:34:41.299 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBl6P5lv7pCExAACgyFS3tWsnorh+23JfVMFQ== 2024-03-23T14:34:41.299 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-23T14:34:41.299 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBo6P5lN9XjERAAY7Doq86QfXJ46LU3H9ET4g== 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBr6P5le9jdDxAAF7YY5jy4+03N9tB79FqgxA== 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.3] 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBu6P5lfyVdDhAA6GSFuGvPYAUCczsnMpXFkQ== 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5lz/YnEhAA4Q/N4Xnmj7rkBGQH0WFU5w== 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.b] 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5lcMUxIhAAA7sdubbg/sFxR2JSLj6knA== 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout:[mds.d] 2024-03-23T14:34:41.300 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5la0WTKRAACwJ5zIvawF/Gsu3pgRkjRQ== 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.4] 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB06P5lxMy+FhAAhO1d5E1ioKlC3bqbkingpA== 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.5] 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB36P5lHl2nFRAA18JrgqJgj6mYykRjhemRvQ== 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.6] 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB66P5lC5OzFBAAeJslkQ08ZxK2fx9yQKsCOA== 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.7] 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQB96P5lyiCSEhAAn+5LSA94oOhlckTfTUd7Lw== 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-03-23T14:34:41.301 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:34:41.302 INFO:teuthology.orchestra.run.smithi183.stdout:[client.2] 2024-03-23T14:34:41.302 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5lIYbHMxAAOYcn/3VZTDW2K3OBKfjxkA== 2024-03-23T14:34:41.302 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2024-03-23T14:34:41.302 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBh6P5lskU4NxAAed0RZpsIuzbux+yi642hsw== 2024-03-23T14:34:41.302 INFO:teuthology.orchestra.run.smithi183.stdout:[client.3] 2024-03-23T14:34:41.302 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQBi6P5lwCRzARAAlFyun7fjX8hB/KsVJhLfWg== 2024-03-23T14:34:41.303 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T14:34:41.321 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T14:34:41.395 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T14:34:41.398 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T14:34:41.452 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T14:34:41.454 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T14:34:41.509 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.512 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.563 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.566 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.617 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.620 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.672 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.674 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.725 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T14:34:41.727 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-23T14:34:41.779 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T14:34:41.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T14:34:41.837 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T14:34:41.839 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-03-23T14:34:41.893 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.896 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.948 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:41.950 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:42.002 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:42.005 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:42.058 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:42.060 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-23T14:34:42.112 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T14:34:42.118 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T14:34:42.173 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T14:34:42.175 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T14:34:42.230 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T14:34:42.232 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T14:34:42.287 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T14:34:42.289 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-23T14:34:42.344 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-23T14:34:42.344 DEBUG:teuthology.orchestra.run.smithi161:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-23T14:34:42.371 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T14:34:42.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-23T14:34:42.495 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-23T14:34:42.523 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T14:34:42.622 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-23T14:34:42.650 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-23T14:34:42.717 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-23T14:34:42.814 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-23T14:34:42.841 DEBUG:teuthology.orchestra.run.smithi161:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:42.844 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-23T14:34:42.896 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-23T14:34:42.897 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-23T14:34:42.897 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-23T14:34:42.900 INFO:tasks.ceph.mon.a:Started 2024-03-23T14:34:42.901 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-23T14:34:42.901 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-23T14:34:42.941 INFO:tasks.ceph.mon.b:Started 2024-03-23T14:34:42.942 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-23T14:34:42.942 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-23T14:34:42.945 INFO:tasks.ceph.mon.c:Started 2024-03-23T14:34:42.945 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-23T14:34:42.945 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-23T14:34:42.945 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-23T14:34:42.948 INFO:tasks.ceph.mgr.y:Started 2024-03-23T14:34:42.948 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-23T14:34:42.948 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-23T14:34:42.950 INFO:tasks.ceph.mgr.x:Started 2024-03-23T14:34:42.951 INFO:tasks.ceph:Setting crush tunables to default 2024-03-23T14:34:42.951 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph osd crush tunables default 2024-03-23T14:34:43.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.057+0000 7f7dafd13640 1 Processor -- start 2024-03-23T14:34:43.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.058+0000 7f7dafd13640 1 -- start start 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.058+0000 7f7dafd13640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.058+0000 7f7dafd13640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da8144130 con 0x7f7da80af730 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.058+0000 7f7dafd13640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da80aff60 con 0x7f7da80ceb10 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.058+0000 7f7dafd13640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7da80ade10 con 0x7f7da8071c80 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.058+0000 7f7dadaaf640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.059+0000 7f7dae2b0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7da80af730 0x7f7da80cc0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:36446/0 (socket says 172.21.15.161:36446) 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.059+0000 7f7dadaaf640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:39536/0 (socket says 172.21.15.161:39536) 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.059+0000 7f7dae2b0640 1 -- 172.21.15.161:0/3388063148 learned_addr learned my addr 172.21.15.161:0/3388063148 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.059+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.059+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.059+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:43.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.059+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T14:34:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.260+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.260+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T14:34:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.260+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.260+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.260+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T14:34:43.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.661+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:43.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.661+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T14:34:43.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.661+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:43.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.661+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:43.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:43.661+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T14:34:44.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:44.462+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:44.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:44.462+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T14:34:44.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:44.462+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:44.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:44.462+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:44.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:44.462+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T14:34:46.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 msgr2=0x7f7da8144800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7d9f7fe640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.161:3300/0 conn(0x7f7da8071c80 0x7f7da8144800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 >> v1:172.21.15.183:6789/0 conn(0x7f7da80ceb10 legacy=0x7f7da8072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 >> v1:172.21.15.161:6789/0 conn(0x7f7da80af730 legacy=0x7f7da80cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7d9f7fe640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7d9f7fe640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8c00af20 con 0x7f7d8c0083e0 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8c00b090 con 0x7f7d8c005920 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T14:34:46.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:46.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:46.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T14:34:46.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T14:34:46.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:46.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:46.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.059+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-23T14:34:46.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:46.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:46.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T14:34:46.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:46.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:46.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T14:34:46.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T14:34:46.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:46.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:46.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.260+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-23T14:34:46.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:46.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:46.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T14:34:46.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:46.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T14:34:46.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:46.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:46.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:46.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T14:34:46.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:46.661+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-23T14:34:47.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dadaaf640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-23T14:34:47.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:47.463+0000 7f7dadaaf640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-23T14:34:48.754 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:48.758+0000 7f1bf5441040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T14:34:48.805 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:48.810+0000 7f1bf5441040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 msgr2=0x7f7d8c007d70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005920 0x7f7d8c007d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 msgr2=0x7f7d8c00a8a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c0083e0 0x7f7d8c00a8a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dae2b0640 1 -- 172.21.15.161:0/3388063148 reap_dead start 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005d60 0x7f7d8c008810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c008d50 0x7f7d8c0091d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7da80cb300 con 0x7f7da80af730 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f7d8c00af20 con 0x7f7d8c008d50 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7d9f7fe640 1 -- 172.21.15.161:0/3388063148 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7d8c00b090 con 0x7f7d8c005d60 2024-03-23T14:34:49.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005d60 0x7f7d8c008810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dae2b0640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c008d50 0x7f7d8c0091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dacaad640 1 -- 172.21.15.161:0/3388063148 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3025437129 0 0) 0x7f7da80cb300 con 0x7f7da80af730 2024-03-23T14:34:49.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dacaad640 1 -- 172.21.15.161:0/3388063148 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7d94003520 con 0x7f7da80af730 2024-03-23T14:34:49.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c008d50 msgr2=0x7f7d8c0091d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:49.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c008d50 0x7f7d8c0091d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 >> v1:172.21.15.161:6789/0 conn(0x7f7da80af730 legacy=0x7f7d8c00d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.059+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3388063148 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d8c00a9e0 con 0x7f7d8c005d60 2024-03-23T14:34:49.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.060+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005d60 0x7f7d8c008810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7da80cef30 tx=0x7f7d9802c050 comp rx=0 tx=0).ready entity=mon.2 client_cookie=401c90982640dc4c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dacaad640 1 -- 172.21.15.161:0/3388063148 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9802ccc0 con 0x7f7d8c005d60 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dacaad640 1 -- 172.21.15.161:0/3388063148 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d9802ce20 con 0x7f7d8c005d60 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dacaad640 1 -- 172.21.15.161:0/3388063148 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d980376a0 con 0x7f7d8c005d60 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dafd13640 1 -- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005d60 msgr2=0x7f7d8c008810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dafd13640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005d60 0x7f7d8c008810 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7da80cef30 tx=0x7f7d9802c050 comp rx=0 tx=0).stop 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dafd13640 1 -- 172.21.15.161:0/3388063148 shutdown_connections 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dafd13640 1 --2- 172.21.15.161:0/3388063148 >> v2:172.21.15.183:3300/0 conn(0x7f7d8c008d50 0x7f7d8c0091d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dafd13640 1 --2- 172.21.15.161:0/3388063148 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c005d60 0x7f7d8c008810 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.061+0000 7f7dafd13640 1 -- 172.21.15.161:0/3388063148 >> 172.21.15.161:0/3388063148 conn(0x7f7da80d0c10 msgr2=0x7f7da80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.062+0000 7f7dafd13640 1 -- 172.21.15.161:0/3388063148 shutdown_connections 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.062+0000 7f7dafd13640 1 -- 172.21.15.161:0/3388063148 wait complete. 2024-03-23T14:34:49.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.063+0000 7f7dafd13640 1 Processor -- start 2024-03-23T14:34:49.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.063+0000 7f7dafd13640 1 -- start start 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dafd13640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d8c005d60 0x7f7da80b4800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dafd13640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c008d50 0x7f7da80b66c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dafd13640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7da80af730 0x7f7da813b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dafd13640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7da80b6c00 con 0x7f7da80af730 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dafd13640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7da80b5a30 con 0x7f7d8c005d60 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dafd13640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7da813bdc0 con 0x7f7d8c008d50 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dad2ae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c008d50 0x7f7da80b66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dadaaf640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d8c005d60 0x7f7da80b4800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dad2ae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c008d50 0x7f7da80b66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:53518/0 (socket says 172.21.15.161:53518) 2024-03-23T14:34:49.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3209188143 learned_addr learned my addr 172.21.15.161:0/3209188143 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dae2b0640 1 --2- 172.21.15.161:0/3209188143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7da80af730 0x7f7da813b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3209188143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d8c005d60 msgr2=0x7f7da80b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3209188143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d8c005d60 0x7f7da80b4800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3209188143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7da80af730 msgr2=0x7f7da813b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3209188143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7da80af730 0x7f7da813b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.064+0000 7f7dad2ae640 1 -- 172.21.15.161:0/3209188143 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d9800f040 con 0x7f7d8c008d50 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.065+0000 7f7dad2ae640 1 --2- 172.21.15.161:0/3209188143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c008d50 0x7f7da80b66c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d98005360 tx=0x7f7d9802e070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.065+0000 7f7d9e7fc640 1 -- 172.21.15.161:0/3209188143 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9802ccc0 con 0x7f7d8c008d50 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.065+0000 7f7d9e7fc640 1 -- 172.21.15.161:0/3209188143 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d98043630 con 0x7f7d8c008d50 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.065+0000 7f7d9e7fc640 1 -- 172.21.15.161:0/3209188143 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d980439e0 con 0x7f7d8c008d50 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.065+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da80b4e20 con 0x7f7d8c008d50 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.065+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da80b53e0 con 0x7f7d8c008d50 2024-03-23T14:34:49.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.065+0000 7f7d9e7fc640 1 -- 172.21.15.161:0/3209188143 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f7d98040030 con 0x7f7d8c008d50 2024-03-23T14:34:49.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.065+0000 7f7d9e7fc640 1 -- 172.21.15.161:0/3209188143 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f7d9804c8f0 con 0x7f7d8c008d50 2024-03-23T14:34:49.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.066+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7da809f100 con 0x7f7d8c008d50 2024-03-23T14:34:49.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.068+0000 7f7d9e7fc640 1 -- 172.21.15.161:0/3209188143 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f7d98032070 con 0x7f7d8c008d50 2024-03-23T14:34:49.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.137+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f7da809f100 con 0x7f7d8c008d50 2024-03-23T14:34:49.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.140+0000 7f7d9e7fc640 1 -- 172.21.15.161:0/3209188143 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f7d9804c440 con 0x7f7d8c008d50 2024-03-23T14:34:49.136 INFO:teuthology.orchestra.run.smithi161.stderr:adjusted tunables profile to default 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c008d50 msgr2=0x7f7da80b66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 --2- 172.21.15.161:0/3209188143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c008d50 0x7f7da80b66c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7d98005360 tx=0x7f7d9802e070 comp rx=0 tx=0).stop 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 shutdown_connections 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 --2- 172.21.15.161:0/3209188143 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7da80af730 0x7f7da813b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 --2- 172.21.15.161:0/3209188143 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d8c008d50 0x7f7da80b66c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 --2- 172.21.15.161:0/3209188143 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d8c005d60 0x7f7da80b4800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 >> 172.21.15.161:0/3209188143 conn(0x7f7da80d0c10 msgr2=0x7f7da80aebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 shutdown_connections 2024-03-23T14:34:49.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:49.141+0000 7f7dafd13640 1 -- 172.21.15.161:0/3209188143 wait complete. 2024-03-23T14:34:49.148 INFO:tasks.ceph:check_enable_crimson: False 2024-03-23T14:34:49.148 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-23T14:34:49.149 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:49.149 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-23T14:34:49.177 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:49.177 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-23T14:34:49.244 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:49.244 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-23T14:34:49.311 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:34:49.311 DEBUG:teuthology.orchestra.run.smithi161:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-23T14:34:49.341 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.346+0000 7f1bf5441040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T14:34:49.379 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:49.379 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-23T14:34:49.395 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.400+0000 7f1bf5441040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T14:34:49.406 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:49.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-23T14:34:49.442 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.446+0000 7f1bf5441040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T14:34:49.469 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:49.469 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-23T14:34:49.498 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.502+0000 7f1bf5441040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T14:34:49.524 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:34:49.524 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-23T14:34:49.534 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:49.539+0000 7f819be73040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-23T14:34:49.542 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.547+0000 7f1bf5441040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T14:34:49.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new b5759fd7-481a-4d08-94da-d5f47464dc44 0 2024-03-23T14:34:49.589 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:49.593+0000 7f819be73040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-23T14:34:49.664 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.668+0000 7f1bf5441040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T14:34:49.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.714+0000 7f3691b7a640 1 Processor -- start 2024-03-23T14:34:49.710 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.714+0000 7f3691b7a640 1 -- start start 2024-03-23T14:34:49.711 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f3691b7a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f368c0ceb10 0x7f368c144800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.711 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f3691b7a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f368c0af730 0x7f368c0cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.711 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f3691b7a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f368c144130 con 0x7f368c071c80 2024-03-23T14:34:49.711 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f3691b7a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f368c0ade10 con 0x7f368c0ceb10 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f3691b7a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f368c0aff60 con 0x7f368c0af730 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f368bfff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f368c0af730 0x7f368c0cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f368bfff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f368c0af730 0x7f368c0cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42510/0 (socket says 172.21.15.183:42510) 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f368bfff640 1 -- 172.21.15.183:0/1065540151 learned_addr learned my addr 172.21.15.183:0/1065540151 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f3683fff640 1 --2- 172.21.15.183:0/1065540151 >> v2:172.21.15.161:3300/0 conn(0x7f368c0ceb10 0x7f368c144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f368bfff640 1 -- 172.21.15.183:0/1065540151 >> v2:172.21.15.161:3300/0 conn(0x7f368c0ceb10 msgr2=0x7f368c144800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f368bfff640 1 --2- 172.21.15.183:0/1065540151 >> v2:172.21.15.161:3300/0 conn(0x7f368c0ceb10 0x7f368c144800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f368bfff640 1 -- 172.21.15.183:0/1065540151 >> v1:172.21.15.161:6789/0 conn(0x7f368c071c80 legacy=0x7f368c072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.712 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.715+0000 7f368bfff640 1 -- 172.21.15.183:0/1065540151 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368c0cb300 con 0x7f368c0af730 2024-03-23T14:34:49.713 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f368bfff640 1 --2- 172.21.15.183:0/1065540151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c0af730 0x7f368c0cc0c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f368c071c10 tx=0x7f368402bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79ef6d4bc425414a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:49.713 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f368affd640 1 -- 172.21.15.183:0/1065540151 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368402cb20 con 0x7f368c0af730 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f368affd640 1 -- 172.21.15.183:0/1065540151 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f368402cc80 con 0x7f368c0af730 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f368affd640 1 -- 172.21.15.183:0/1065540151 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36840343f0 con 0x7f368c0af730 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f3691b7a640 1 -- 172.21.15.183:0/1065540151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c0af730 msgr2=0x7f368c0cc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f3691b7a640 1 --2- 172.21.15.183:0/1065540151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c0af730 0x7f368c0cc0c0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f368c071c10 tx=0x7f368402bf80 comp rx=0 tx=0).stop 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f3691b7a640 1 -- 172.21.15.183:0/1065540151 shutdown_connections 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f3691b7a640 1 --2- 172.21.15.183:0/1065540151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c0af730 0x7f368c0cc0c0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f3691b7a640 1 --2- 172.21.15.183:0/1065540151 >> v2:172.21.15.161:3300/0 conn(0x7f368c0ceb10 0x7f368c144800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.717+0000 7f3691b7a640 1 -- 172.21.15.183:0/1065540151 >> 172.21.15.183:0/1065540151 conn(0x7f368c0d0c10 msgr2=0x7f368c0d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.718+0000 7f3691b7a640 1 -- 172.21.15.183:0/1065540151 shutdown_connections 2024-03-23T14:34:49.714 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.718+0000 7f3691b7a640 1 -- 172.21.15.183:0/1065540151 wait complete. 2024-03-23T14:34:49.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.719+0000 7f3691b7a640 1 Processor -- start 2024-03-23T14:34:49.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.719+0000 7f3691b7a640 1 -- start start 2024-03-23T14:34:49.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.719+0000 7f3691b7a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c071c80 0x7f368c15a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.719+0000 7f3691b7a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0af730 0x7f368c140f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.719+0000 7f3691b7a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0ceb10 0x7f368c141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.719+0000 7f3691b7a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f368c1f1d00 con 0x7f368c0af730 2024-03-23T14:34:49.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.719+0000 7f3691b7a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f368c15abf0 con 0x7f368c071c80 2024-03-23T14:34:49.717 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.719+0000 7f3691b7a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f368c1f1150 con 0x7f368c0ceb10 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368b7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c071c80 0x7f368c15a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368b7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c071c80 0x7f368c15a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42514/0 (socket says 172.21.15.183:42514) 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368bfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0ceb10 0x7f368c141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368b7fe640 1 -- 172.21.15.183:0/866321146 learned_addr learned my addr 172.21.15.183:0/866321146 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3683fff640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0af730 0x7f368c140f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368bfff640 1 -- 172.21.15.183:0/866321146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c071c80 msgr2=0x7f368c15a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368bfff640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c071c80 0x7f368c15a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368bfff640 1 -- 172.21.15.183:0/866321146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0af730 msgr2=0x7f368c140f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368bfff640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0af730 0x7f368c140f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368bfff640 1 -- 172.21.15.183:0/866321146 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3684009b30 con 0x7f368c0ceb10 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368b7fe640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c071c80 0x7f368c15a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3683fff640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0af730 0x7f368c140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:49.718 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f368bfff640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0ceb10 0x7f368c141470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3684032000 tx=0x7f3684004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:49.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3688ff9640 1 -- 172.21.15.183:0/866321146 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3684009510 con 0x7f368c0ceb10 2024-03-23T14:34:49.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3688ff9640 1 -- 172.21.15.183:0/866321146 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3684007e00 con 0x7f368c0ceb10 2024-03-23T14:34:49.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f368c1f2120 con 0x7f368c0ceb10 2024-03-23T14:34:49.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3688ff9640 1 -- 172.21.15.183:0/866321146 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f368403e690 con 0x7f368c0ceb10 2024-03-23T14:34:49.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f368c1f1640 con 0x7f368c0ceb10 2024-03-23T14:34:49.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3688ff9640 1 -- 172.21.15.183:0/866321146 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f368403e7f0 con 0x7f368c0ceb10 2024-03-23T14:34:49.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.720+0000 7f3688ff9640 1 -- 172.21.15.183:0/866321146 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7f36840488b0 con 0x7f368c0ceb10 2024-03-23T14:34:49.719 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.721+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f368c09f100 con 0x7f368c0ceb10 2024-03-23T14:34:49.720 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.725+0000 7f3688ff9640 1 -- 172.21.15.183:0/866321146 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3684043020 con 0x7f368c0ceb10 2024-03-23T14:34:49.738 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.743+0000 7f1bf5441040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T14:34:49.785 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.790+0000 7f1bf5441040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T14:34:49.793 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.797+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b5759fd7-481a-4d08-94da-d5f47464dc44", "id": 0} v 0) -- 0x7f368c04f550 con 0x7f368c0ceb10 2024-03-23T14:34:49.796 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.801+0000 7f3688ff9640 1 -- 172.21.15.183:0/866321146 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b5759fd7-481a-4d08-94da-d5f47464dc44", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7f3684048400 con 0x7f368c0ceb10 2024-03-23T14:34:49.796 INFO:teuthology.orchestra.run.smithi183.stdout:0 2024-03-23T14:34:49.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.801+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0ceb10 msgr2=0x7f368c141470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.801+0000 7f3691b7a640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0ceb10 0x7f368c141470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3684032000 tx=0x7f3684004a40 comp rx=0 tx=0).stop 2024-03-23T14:34:49.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.801+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 shutdown_connections 2024-03-23T14:34:49.797 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.801+0000 7f3691b7a640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0ceb10 0x7f368c141470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.801+0000 7f3691b7a640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0af730 0x7f368c140f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.801+0000 7f3691b7a640 1 --2- 172.21.15.183:0/866321146 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c071c80 0x7f368c15a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.801+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 >> 172.21.15.183:0/866321146 conn(0x7f368c0d0c10 msgr2=0x7f368c0cf7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:49.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.802+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 shutdown_connections 2024-03-23T14:34:49.798 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.802+0000 7f3691b7a640 1 -- 172.21.15.183:0/866321146 wait complete. 2024-03-23T14:34:49.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 6c8004b2-58d7-46c3-a737-d376a959fde1 1 2024-03-23T14:34:49.842 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.847+0000 7f1bf5441040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T14:34:49.894 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:49.896+0000 7f1bf5441040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T14:34:49.907 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.912+0000 7f5c0fad4640 1 Processor -- start 2024-03-23T14:34:49.908 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.912+0000 7f5c0fad4640 1 -- start start 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0fad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c08071c80 0x7f5c08144800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0fad4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5c080af730 0x7f5c080cc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0fad4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c08144130 con 0x7f5c080ceb10 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0fad4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f5c080ade10 con 0x7f5c080af730 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0fad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5c080aff60 con 0x7f5c08071c80 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0e071640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5c080af730 0x7f5c080cc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0d870640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c08071c80 0x7f5c08144800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0d06f640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f5c080ceb10 0x7f5c08072060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:47030/0 (socket says 172.21.15.183:47030) 2024-03-23T14:34:49.909 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0e071640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5c080af730 0x7f5c080cc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42516/0 (socket says 172.21.15.183:42516) 2024-03-23T14:34:49.910 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0d06f640 1 -- 172.21.15.183:0/74713323 learned_addr learned my addr 172.21.15.183:0/74713323 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-23T14:34:49.910 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0d870640 1 -- 172.21.15.183:0/74713323 >> v2:172.21.15.183:3300/0 conn(0x7f5c080af730 msgr2=0x7f5c080cc0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:49.910 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0d870640 1 --2- 172.21.15.183:0/74713323 >> v2:172.21.15.183:3300/0 conn(0x7f5c080af730 0x7f5c080cc0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.910 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0d870640 1 -- 172.21.15.183:0/74713323 >> v1:172.21.15.183:6789/0 conn(0x7f5c080ceb10 legacy=0x7f5c08072060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.910 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0d870640 1 -- 172.21.15.183:0/74713323 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c080cb300 con 0x7f5c08071c80 2024-03-23T14:34:49.910 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0c86e640 1 -- 172.21.15.183:0/74713323 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3001162168 0 0) 0x7f5c08144130 con 0x7f5c080ceb10 2024-03-23T14:34:49.910 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.913+0000 7f5c0e071640 1 --2- 172.21.15.183:0/74713323 >> v2:172.21.15.183:3300/0 conn(0x7f5c080af730 0x7f5c080cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:49.911 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.914+0000 7f5c0d870640 1 --2- 172.21.15.183:0/74713323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c08071c80 0x7f5c08144800 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400ba40 tx=0x7f5c0402e080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f3de9a476757b38 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.914+0000 7f5c0c86e640 1 -- 172.21.15.183:0/74713323 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0402ee30 con 0x7f5c08071c80 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.914+0000 7f5c0c86e640 1 -- 172.21.15.183:0/74713323 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c04036440 con 0x7f5c08071c80 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.914+0000 7f5c0c86e640 1 -- 172.21.15.183:0/74713323 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c040367d0 con 0x7f5c08071c80 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.915+0000 7f5c0fad4640 1 -- 172.21.15.183:0/74713323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c08071c80 msgr2=0x7f5c08144800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.915+0000 7f5c0fad4640 1 --2- 172.21.15.183:0/74713323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c08071c80 0x7f5c08144800 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400ba40 tx=0x7f5c0402e080 comp rx=0 tx=0).stop 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.915+0000 7f5c0fad4640 1 -- 172.21.15.183:0/74713323 shutdown_connections 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.915+0000 7f5c0fad4640 1 --2- 172.21.15.183:0/74713323 >> v2:172.21.15.183:3300/0 conn(0x7f5c080af730 0x7f5c080cc0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.915+0000 7f5c0fad4640 1 --2- 172.21.15.183:0/74713323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c08071c80 0x7f5c08144800 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.915+0000 7f5c0fad4640 1 -- 172.21.15.183:0/74713323 >> 172.21.15.183:0/74713323 conn(0x7f5c080d0c10 msgr2=0x7f5c080d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.915+0000 7f5c0fad4640 1 -- 172.21.15.183:0/74713323 shutdown_connections 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.915+0000 7f5c0fad4640 1 -- 172.21.15.183:0/74713323 wait complete. 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.916+0000 7f5c0fad4640 1 Processor -- start 2024-03-23T14:34:49.912 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.916+0000 7f5c0fad4640 1 -- start start 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0fad4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c08071c80 0x7f5c0813c5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0fad4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c080af730 0x7f5c080b3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0fad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c080ceb10 0x7f5c080b3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0fad4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0815bda0 con 0x7f5c080af730 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0fad4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0815c080 con 0x7f5c08071c80 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0fad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5c0815c360 con 0x7f5c080ceb10 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0d870640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c08071c80 0x7f5c0813c5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0e071640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c080ceb10 0x7f5c080b3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0d870640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c08071c80 0x7f5c0813c5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42526/0 (socket says 172.21.15.183:42526) 2024-03-23T14:34:49.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0d870640 1 -- 172.21.15.183:0/3644770858 learned_addr learned my addr 172.21.15.183:0/3644770858 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0d06f640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c080af730 0x7f5c080b3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0e071640 1 -- 172.21.15.183:0/3644770858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c08071c80 msgr2=0x7f5c0813c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0e071640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c08071c80 0x7f5c0813c5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0e071640 1 -- 172.21.15.183:0/3644770858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c080af730 msgr2=0x7f5c080b3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0e071640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c080af730 0x7f5c080b3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0e071640 1 -- 172.21.15.183:0/3644770858 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c00009a70 con 0x7f5c080ceb10 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0d870640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c08071c80 0x7f5c0813c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0e071640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c080ceb10 0x7f5c080b3d40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000efc0 tx=0x7f5c0000c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5bfe7fc640 1 -- 172.21.15.183:0/3644770858 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0000cd30 con 0x7f5c080ceb10 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5bfe7fc640 1 -- 172.21.15.183:0/3644770858 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c0000f040 con 0x7f5c080ceb10 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0400b6a0 con 0x7f5c080ceb10 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.917+0000 7f5bfe7fc640 1 -- 172.21.15.183:0/3644770858 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c00003bd0 con 0x7f5c080ceb10 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.918+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0815c920 con 0x7f5c080ceb10 2024-03-23T14:34:49.916 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.918+0000 7f5bfe7fc640 1 -- 172.21.15.183:0/3644770858 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5c00014480 con 0x7f5c080ceb10 2024-03-23T14:34:49.917 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.918+0000 7f5bfe7fc640 1 -- 172.21.15.183:0/3644770858 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f5c0001dcf0 con 0x7f5c080ceb10 2024-03-23T14:34:49.917 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.918+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c0815cb10 con 0x7f5c080ceb10 2024-03-23T14:34:49.917 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.920+0000 7f5bfe7fc640 1 -- 172.21.15.183:0/3644770858 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f5c00018030 con 0x7f5c080ceb10 2024-03-23T14:34:49.989 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.993+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "6c8004b2-58d7-46c3-a737-d376a959fde1", "id": 1} v 0) -- 0x7f5c0809f100 con 0x7f5c080ceb10 2024-03-23T14:34:49.992 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.997+0000 7f5bfe7fc640 1 -- 172.21.15.183:0/3644770858 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6c8004b2-58d7-46c3-a737-d376a959fde1", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f5c0001d410 con 0x7f5c080ceb10 2024-03-23T14:34:49.992 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-23T14:34:49.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.997+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c080ceb10 msgr2=0x7f5c080b3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:49.993 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.997+0000 7f5c0fad4640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c080ceb10 0x7f5c080b3d40 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c0000efc0 tx=0x7f5c0000c490 comp rx=0 tx=0).stop 2024-03-23T14:34:49.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.998+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 shutdown_connections 2024-03-23T14:34:49.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.998+0000 7f5c0fad4640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5c080ceb10 0x7f5c080b3d40 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.998+0000 7f5c0fad4640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5c080af730 0x7f5c080b3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.998+0000 7f5c0fad4640 1 --2- 172.21.15.183:0/3644770858 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5c08071c80 0x7f5c0813c5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:49.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.998+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 >> 172.21.15.183:0/3644770858 conn(0x7f5c080d0c10 msgr2=0x7f5c080cf790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:49.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.998+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 shutdown_connections 2024-03-23T14:34:49.994 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:49.998+0000 7f5c0fad4640 1 -- 172.21.15.183:0/3644770858 wait complete. 2024-03-23T14:34:50.004 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new cb73706e-1fab-4f91-92c3-91c131ae2240 2 2024-03-23T14:34:50.141 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.145+0000 7f819be73040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-23T14:34:50.143 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.148+0000 7fc777a1c640 1 Processor -- start 2024-03-23T14:34:50.144 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.148+0000 7fc777a1c640 1 -- start start 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc777a1c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc77015bec0 0x7fc7700b2a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc777a1c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc7700b2f60 0x7fc7700cf910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc777a1c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7700cfe50 0x7fc7700d1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc777a1c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc7700cde90 con 0x7fc7700b2f60 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc777a1c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc7700d02a0 con 0x7fc77015bec0 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc777a1c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc7700cdbb0 con 0x7fc7700cfe50 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc7757b8640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc77015bec0 0x7fc7700b2a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc775fb9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7700cfe50 0x7fc7700d1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.146 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc7757b8640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc77015bec0 0x7fc7700b2a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42534/0 (socket says 172.21.15.183:42534) 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc775fb9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7700cfe50 0x7fc7700d1ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:38618/0 (socket says 172.21.15.183:38618) 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc7757b8640 1 -- 172.21.15.183:0/2536993580 learned_addr learned my addr 172.21.15.183:0/2536993580 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.149+0000 7fc774fb7640 1 --2- 172.21.15.183:0/2536993580 >> v2:172.21.15.161:3300/0 conn(0x7fc7700b2f60 0x7fc7700cf910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc7757b8640 1 -- 172.21.15.183:0/2536993580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7700cfe50 msgr2=0x7fc7700d1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc7757b8640 1 --2- 172.21.15.183:0/2536993580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7700cfe50 0x7fc7700d1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc7757b8640 1 -- 172.21.15.183:0/2536993580 >> v2:172.21.15.161:3300/0 conn(0x7fc7700b2f60 msgr2=0x7fc7700cf910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc7757b8640 1 --2- 172.21.15.183:0/2536993580 >> v2:172.21.15.161:3300/0 conn(0x7fc7700b2f60 0x7fc7700cf910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc7757b8640 1 -- 172.21.15.183:0/2536993580 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7700b1f40 con 0x7fc77015bec0 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc775fb9640 1 --2- 172.21.15.183:0/2536993580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7700cfe50 0x7fc7700d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc7757b8640 1 --2- 172.21.15.183:0/2536993580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc77015bec0 0x7fc7700b2a20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc77006e530 tx=0x7fc76002b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58344870402fe318 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc767fff640 1 -- 172.21.15.183:0/2536993580 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760031870 con 0x7fc77015bec0 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc767fff640 1 -- 172.21.15.183:0/2536993580 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7600319d0 con 0x7fc77015bec0 2024-03-23T14:34:50.147 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.150+0000 7fc767fff640 1 -- 172.21.15.183:0/2536993580 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc760031d60 con 0x7fc77015bec0 2024-03-23T14:34:50.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 -- 172.21.15.183:0/2536993580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc77015bec0 msgr2=0x7fc7700b2a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 --2- 172.21.15.183:0/2536993580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc77015bec0 0x7fc7700b2a20 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc77006e530 tx=0x7fc76002b910 comp rx=0 tx=0).stop 2024-03-23T14:34:50.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 -- 172.21.15.183:0/2536993580 shutdown_connections 2024-03-23T14:34:50.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 --2- 172.21.15.183:0/2536993580 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7700cfe50 0x7fc7700d1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.150 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 --2- 172.21.15.183:0/2536993580 >> v2:172.21.15.161:3300/0 conn(0x7fc7700b2f60 0x7fc7700cf910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 --2- 172.21.15.183:0/2536993580 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc77015bec0 0x7fc7700b2a20 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 -- 172.21.15.183:0/2536993580 >> 172.21.15.183:0/2536993580 conn(0x7fc77009aa40 msgr2=0x7fc77009ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 -- 172.21.15.183:0/2536993580 shutdown_connections 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.151+0000 7fc777a1c640 1 -- 172.21.15.183:0/2536993580 wait complete. 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.152+0000 7fc777a1c640 1 Processor -- start 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc777a1c640 1 -- start start 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc777a1c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7700b2f60 0x7fc7701527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc777a1c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7700cfe50 0x7fc7700c9e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc777a1c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc77015bec0 0x7fc770156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc777a1c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc77006ce40 con 0x7fc7700b2f60 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc777a1c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc770159420 con 0x7fc7700cfe50 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc777a1c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc77006c230 con 0x7fc77015bec0 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc774fb7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7700cfe50 0x7fc7700c9e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.151 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc775fb9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc77015bec0 0x7fc770156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc774fb7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7700cfe50 0x7fc7700c9e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42540/0 (socket says 172.21.15.183:42540) 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc774fb7640 1 -- 172.21.15.183:0/286460476 learned_addr learned my addr 172.21.15.183:0/286460476 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.153+0000 7fc7757b8640 1 --2- 172.21.15.183:0/286460476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7700b2f60 0x7fc7701527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc775fb9640 1 -- 172.21.15.183:0/286460476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7700cfe50 msgr2=0x7fc7700c9e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc775fb9640 1 --2- 172.21.15.183:0/286460476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7700cfe50 0x7fc7700c9e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc775fb9640 1 -- 172.21.15.183:0/286460476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7700b2f60 msgr2=0x7fc7701527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc775fb9640 1 --2- 172.21.15.183:0/286460476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7700b2f60 0x7fc7701527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc775fb9640 1 -- 172.21.15.183:0/286460476 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc768009a70 con 0x7fc77015bec0 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc775fb9640 1 --2- 172.21.15.183:0/286460476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc77015bec0 0x7fc770156d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc768009e50 tx=0x7fc76800c490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc765ffb640 1 -- 172.21.15.183:0/286460476 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc76800cd90 con 0x7fc77015bec0 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc765ffb640 1 -- 172.21.15.183:0/286460476 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc76800f040 con 0x7fc77015bec0 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc765ffb640 1 -- 172.21.15.183:0/286460476 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc768003bd0 con 0x7fc77015bec0 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7600094d0 con 0x7fc77015bec0 2024-03-23T14:34:50.152 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc765ffb640 1 -- 172.21.15.183:0/286460476 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc768003d30 con 0x7fc77015bec0 2024-03-23T14:34:50.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc77015c420 con 0x7fc77015bec0 2024-03-23T14:34:50.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.154+0000 7fc765ffb640 1 -- 172.21.15.183:0/286460476 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fc768014c60 con 0x7fc77015bec0 2024-03-23T14:34:50.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.155+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc77006cff0 con 0x7fc77015bec0 2024-03-23T14:34:50.153 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.157+0000 7fc765ffb640 1 -- 172.21.15.183:0/286460476 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc77006cff0 con 0x7fc77015bec0 2024-03-23T14:34:50.197 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.201+0000 7f819be73040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-23T14:34:50.225 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.229+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "cb73706e-1fab-4f91-92c3-91c131ae2240", "id": 2} v 0) -- 0x7fc77009fcd0 con 0x7fc77015bec0 2024-03-23T14:34:50.228 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.233+0000 7fc765ffb640 1 -- 172.21.15.183:0/286460476 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cb73706e-1fab-4f91-92c3-91c131ae2240", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fc768018030 con 0x7fc77015bec0 2024-03-23T14:34:50.228 INFO:teuthology.orchestra.run.smithi183.stdout:2 2024-03-23T14:34:50.229 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc77015bec0 msgr2=0x7fc770156d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 --2- 172.21.15.183:0/286460476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc77015bec0 0x7fc770156d40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc768009e50 tx=0x7fc76800c490 comp rx=0 tx=0).stop 2024-03-23T14:34:50.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 shutdown_connections 2024-03-23T14:34:50.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 --2- 172.21.15.183:0/286460476 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc77015bec0 0x7fc770156d40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 --2- 172.21.15.183:0/286460476 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7700cfe50 0x7fc7700c9e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 --2- 172.21.15.183:0/286460476 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7700b2f60 0x7fc7701527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 >> 172.21.15.183:0/286460476 conn(0x7fc77009aa40 msgr2=0x7fc7700670e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 shutdown_connections 2024-03-23T14:34:50.230 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.234+0000 7fc777a1c640 1 -- 172.21.15.183:0/286460476 wait complete. 2024-03-23T14:34:50.240 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 37ac7a35-54f4-43ca-8fe1-cbee435e2de6 3 2024-03-23T14:34:50.243 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.247+0000 7f819be73040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-23T14:34:50.288 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:50.293+0000 7f1bf5441040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T14:34:50.297 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.302+0000 7f819be73040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-23T14:34:50.341 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.345+0000 7f819be73040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-23T14:34:50.342 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.345+0000 7f4dc8925640 1 Processor -- start 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.345+0000 7f4dc8925640 1 -- start start 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc8925640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4dc40cf2d0 0x7f4dc40d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc8925640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40d0630 0x7f4dc40cc210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc8925640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4dc40d0ad0 con 0x7f4dc40c6310 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc8925640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f4dc40cb450 con 0x7f4dc40cf2d0 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc8925640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc40cb730 con 0x7f4dc40d0630 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc1d9b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4dc40cf2d0 0x7f4dc40d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc2d9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40d0630 0x7f4dc40cc210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc1d9b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4dc40cf2d0 0x7f4dc40d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42548/0 (socket says 172.21.15.183:42548) 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc2d9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40d0630 0x7f4dc40cc210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:38624/0 (socket says 172.21.15.183:38624) 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.346+0000 7f4dc1d9b640 1 -- 172.21.15.183:0/3866212955 learned_addr learned my addr 172.21.15.183:0/3866212955 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.347+0000 7f4dc2d9d640 1 -- 172.21.15.183:0/3866212955 >> v2:172.21.15.183:3300/0 conn(0x7f4dc40cf2d0 msgr2=0x7f4dc40d00f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:50.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.347+0000 7f4dc2d9d640 1 --2- 172.21.15.183:0/3866212955 >> v2:172.21.15.183:3300/0 conn(0x7f4dc40cf2d0 0x7f4dc40d00f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.344 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.347+0000 7f4dc2d9d640 1 -- 172.21.15.183:0/3866212955 >> v1:172.21.15.161:6789/0 conn(0x7f4dc40c6310 legacy=0x7f4dc40c66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.344 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.347+0000 7f4dc159a640 1 -- 172.21.15.183:0/3866212955 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1387932783 0 0) 0x7f4dc40d0ad0 con 0x7f4dc40c6310 2024-03-23T14:34:50.344 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.347+0000 7f4dc2d9d640 1 -- 172.21.15.183:0/3866212955 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dc40cba10 con 0x7f4dc40d0630 2024-03-23T14:34:50.344 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.347+0000 7f4dc1d9b640 1 --2- 172.21.15.183:0/3866212955 >> v2:172.21.15.183:3300/0 conn(0x7f4dc40cf2d0 0x7f4dc40d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:50.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.347+0000 7f4dc2d9d640 1 --2- 172.21.15.183:0/3866212955 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40d0630 0x7f4dc40cc210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009d90 tx=0x7f4dbc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31cb2f35dde4fdfc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc159a640 1 -- 172.21.15.183:0/3866212955 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc02cc50 con 0x7f4dc40d0630 2024-03-23T14:34:50.345 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc159a640 1 -- 172.21.15.183:0/3866212955 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dbc02cdb0 con 0x7f4dc40d0630 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc159a640 1 -- 172.21.15.183:0/3866212955 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dbc034660 con 0x7f4dc40d0630 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc8925640 1 -- 172.21.15.183:0/3866212955 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40d0630 msgr2=0x7f4dc40cc210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc8925640 1 --2- 172.21.15.183:0/3866212955 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40d0630 0x7f4dc40cc210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4dbc009d90 tx=0x7f4dbc02bea0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc8925640 1 -- 172.21.15.183:0/3866212955 shutdown_connections 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc8925640 1 --2- 172.21.15.183:0/3866212955 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40d0630 0x7f4dc40cc210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc8925640 1 --2- 172.21.15.183:0/3866212955 >> v2:172.21.15.183:3300/0 conn(0x7f4dc40cf2d0 0x7f4dc40d00f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc8925640 1 -- 172.21.15.183:0/3866212955 >> 172.21.15.183:0/3866212955 conn(0x7f4dc40d0c60 msgr2=0x7f4dc40d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc8925640 1 -- 172.21.15.183:0/3866212955 shutdown_connections 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.348+0000 7f4dc8925640 1 -- 172.21.15.183:0/3866212955 wait complete. 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc8925640 1 Processor -- start 2024-03-23T14:34:50.346 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc8925640 1 -- start start 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc8925640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4dc40c6310 0x7f4dc415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc8925640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40cf2d0 0x7f4dc413c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc8925640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4dc40d0630 0x7f4dc40c9e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc8925640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc40c9990 con 0x7f4dc40d0630 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc8925640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4dc40c8d80 con 0x7f4dc40c6310 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc8925640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4dc40c9060 con 0x7f4dc40cf2d0 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc259c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4dc40c6310 0x7f4dc415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.350+0000 7f4dc1d9b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40cf2d0 0x7f4dc413c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc1d9b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40cf2d0 0x7f4dc413c030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:38638/0 (socket says 172.21.15.183:38638) 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc259c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4dc40c6310 0x7f4dc415a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42560/0 (socket says 172.21.15.183:42560) 2024-03-23T14:34:50.349 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc1d9b640 1 -- 172.21.15.183:0/2632230747 learned_addr learned my addr 172.21.15.183:0/2632230747 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc2d9d640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4dc40d0630 0x7f4dc40c9e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc259c640 1 -- 172.21.15.183:0/2632230747 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40cf2d0 msgr2=0x7f4dc413c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc259c640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40cf2d0 0x7f4dc413c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc259c640 1 -- 172.21.15.183:0/2632230747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4dc40d0630 msgr2=0x7f4dc40c9e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc259c640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4dc40d0630 0x7f4dc40c9e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc259c640 1 -- 172.21.15.183:0/2632230747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db80094d0 con 0x7f4dc40c6310 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc1d9b640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40cf2d0 0x7f4dc413c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc259c640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4dc40c6310 0x7f4dc415a9d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c450 tx=0x7f4db400c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4db2ffd640 1 -- 172.21.15.183:0/2632230747 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db40035f0 con 0x7f4dc40c6310 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4db2ffd640 1 -- 172.21.15.183:0/2632230747 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4db4003750 con 0x7f4dc40c6310 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dbc009a70 con 0x7f4dc40c6310 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4db2ffd640 1 -- 172.21.15.183:0/2632230747 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db4003b00 con 0x7f4dc40c6310 2024-03-23T14:34:50.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dc40c9620 con 0x7f4dc40c6310 2024-03-23T14:34:50.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4db2ffd640 1 -- 172.21.15.183:0/2632230747 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f4db4003c60 con 0x7f4dc40c6310 2024-03-23T14:34:50.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.351+0000 7f4db2ffd640 1 -- 172.21.15.183:0/2632230747 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f4db4012890 con 0x7f4dc40c6310 2024-03-23T14:34:50.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.352+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dc406dc40 con 0x7f4dc40c6310 2024-03-23T14:34:50.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.354+0000 7f4db2ffd640 1 -- 172.21.15.183:0/2632230747 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f4db4017070 con 0x7f4dc40c6310 2024-03-23T14:34:50.423 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.425+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "37ac7a35-54f4-43ca-8fe1-cbee435e2de6", "id": 3} v 0) -- 0x7f4dc40736a0 con 0x7f4dc40c6310 2024-03-23T14:34:50.424 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.429+0000 7f4db2ffd640 1 -- 172.21.15.183:0/2632230747 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "37ac7a35-54f4-43ca-8fe1-cbee435e2de6", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f4db4012570 con 0x7f4dc40c6310 2024-03-23T14:34:50.424 INFO:teuthology.orchestra.run.smithi183.stdout:3 2024-03-23T14:34:50.425 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.429+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4dc40c6310 msgr2=0x7f4dc415a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.425 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.429+0000 7f4dc8925640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4dc40c6310 0x7f4dc415a9d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4db400c450 tx=0x7f4db400c910 comp rx=0 tx=0).stop 2024-03-23T14:34:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.429+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 shutdown_connections 2024-03-23T14:34:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.429+0000 7f4dc8925640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4dc40d0630 0x7f4dc40c9e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.429+0000 7f4dc8925640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4dc40cf2d0 0x7f4dc413c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.429+0000 7f4dc8925640 1 --2- 172.21.15.183:0/2632230747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4dc40c6310 0x7f4dc415a9d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.429+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 >> 172.21.15.183:0/2632230747 conn(0x7f4dc40d0c60 msgr2=0x7f4dc40724d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.430+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 shutdown_connections 2024-03-23T14:34:50.426 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.430+0000 7f4dc8925640 1 -- 172.21.15.183:0/2632230747 wait complete. 2024-03-23T14:34:50.426 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:50.431+0000 7f1bf5441040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T14:34:50.435 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a 4 2024-03-23T14:34:50.457 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.460+0000 7f819be73040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-23T14:34:50.526 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.529+0000 7f819be73040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-23T14:34:50.536 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.540+0000 7f43618eb640 1 Processor -- start 2024-03-23T14:34:50.536 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.540+0000 7f43618eb640 1 -- start start 2024-03-23T14:34:50.538 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f43618eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c06b080 0x7f435c06f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.538 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f43618eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f435c0b01c0 0x7f435c06b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.538 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f43618eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f435c06b9a0 0x7f435c06dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.538 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f43618eb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f435c072620 con 0x7f435c06b9a0 2024-03-23T14:34:50.538 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f43618eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f435c0c7b00 con 0x7f435c0b01c0 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f43618eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f435c0c5e50 con 0x7f435c06b080 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f435affd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c06b080 0x7f435c06f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f4353fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f435c0b01c0 0x7f435c06b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f4353fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f435c0b01c0 0x7f435c06b460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42568/0 (socket says 172.21.15.183:42568) 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f435affd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c06b080 0x7f435c06f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:38642/0 (socket says 172.21.15.183:38642) 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f4353fff640 1 -- 172.21.15.183:0/2913284116 learned_addr learned my addr 172.21.15.183:0/2913284116 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f435b7fe640 1 --2- 172.21.15.183:0/2913284116 >> v2:172.21.15.161:3300/0 conn(0x7f435c06b9a0 0x7f435c06dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f435affd640 1 -- 172.21.15.183:0/2913284116 >> v2:172.21.15.183:3300/0 conn(0x7f435c0b01c0 msgr2=0x7f435c06b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f435affd640 1 --2- 172.21.15.183:0/2913284116 >> v2:172.21.15.183:3300/0 conn(0x7f435c0b01c0 0x7f435c06b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f435affd640 1 -- 172.21.15.183:0/2913284116 >> v2:172.21.15.161:3300/0 conn(0x7f435c06b9a0 msgr2=0x7f435c06dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f435affd640 1 --2- 172.21.15.183:0/2913284116 >> v2:172.21.15.161:3300/0 conn(0x7f435c06b9a0 0x7f435c06dcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f435affd640 1 -- 172.21.15.183:0/2913284116 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c0c7820 con 0x7f435c06b080 2024-03-23T14:34:50.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.541+0000 7f4353fff640 1 --2- 172.21.15.183:0/2913284116 >> v2:172.21.15.183:3300/0 conn(0x7f435c0b01c0 0x7f435c06b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:50.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.542+0000 7f435affd640 1 --2- 172.21.15.183:0/2913284116 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c06b080 0x7f435c06f550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f435c067c30 tx=0x7f434c02b710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7801d92da2ad14a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.542+0000 7f435a7fc640 1 -- 172.21.15.183:0/2913284116 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c030710 con 0x7f435c06b080 2024-03-23T14:34:50.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.542+0000 7f435a7fc640 1 -- 172.21.15.183:0/2913284116 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f434c02e050 con 0x7f435c06b080 2024-03-23T14:34:50.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.542+0000 7f435a7fc640 1 -- 172.21.15.183:0/2913284116 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c030a90 con 0x7f435c06b080 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 -- 172.21.15.183:0/2913284116 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c06b080 msgr2=0x7f435c06f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 --2- 172.21.15.183:0/2913284116 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c06b080 0x7f435c06f550 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f435c067c30 tx=0x7f434c02b710 comp rx=0 tx=0).stop 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 -- 172.21.15.183:0/2913284116 shutdown_connections 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 --2- 172.21.15.183:0/2913284116 >> v2:172.21.15.161:3300/0 conn(0x7f435c06b9a0 0x7f435c06dcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 --2- 172.21.15.183:0/2913284116 >> v2:172.21.15.183:3300/0 conn(0x7f435c0b01c0 0x7f435c06b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 --2- 172.21.15.183:0/2913284116 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c06b080 0x7f435c06f550 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 -- 172.21.15.183:0/2913284116 >> 172.21.15.183:0/2913284116 conn(0x7f435c0d0c60 msgr2=0x7f435c0d1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 -- 172.21.15.183:0/2913284116 shutdown_connections 2024-03-23T14:34:50.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.543+0000 7f43618eb640 1 -- 172.21.15.183:0/2913284116 wait complete. 2024-03-23T14:34:50.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.544+0000 7f43618eb640 1 Processor -- start 2024-03-23T14:34:50.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.544+0000 7f43618eb640 1 -- start start 2024-03-23T14:34:50.542 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f1bf5441040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f43618eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c06b080 0x7f435c1f30e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f43618eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f435c06b9a0 0x7f435c0cf990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f43618eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c0b01c0 0x7f435c0cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f43618eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f435c157630 con 0x7f435c06b080 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f43618eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f435c1f7080 con 0x7f435c06b9a0 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f43618eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f435c1f7360 con 0x7f435c0b01c0 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f4353fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f435c06b9a0 0x7f435c0cf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435b7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c0b01c0 0x7f435c0cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435b7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c0b01c0 0x7f435c0cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:38656/0 (socket says 172.21.15.183:38656) 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f4353fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f435c06b9a0 0x7f435c0cf990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42576/0 (socket says 172.21.15.183:42576) 2024-03-23T14:34:50.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435b7fe640 1 -- 172.21.15.183:0/1459798885 learned_addr learned my addr 172.21.15.183:0/1459798885 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435affd640 1 --2- 172.21.15.183:0/1459798885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c06b080 0x7f435c1f30e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435b7fe640 1 -- 172.21.15.183:0/1459798885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f435c06b9a0 msgr2=0x7f435c0cf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435b7fe640 1 --2- 172.21.15.183:0/1459798885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f435c06b9a0 0x7f435c0cf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435b7fe640 1 -- 172.21.15.183:0/1459798885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c06b080 msgr2=0x7f435c1f30e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435b7fe640 1 --2- 172.21.15.183:0/1459798885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c06b080 0x7f435c1f30e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.545+0000 7f435b7fe640 1 -- 172.21.15.183:0/1459798885 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43440094d0 con 0x7f435c0b01c0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.546+0000 7f435b7fe640 1 --2- 172.21.15.183:0/1459798885 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c0b01c0 0x7f435c0cfed0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f435400eed0 tx=0x7f435400b480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.546+0000 7f4358ff9640 1 -- 172.21.15.183:0/1459798885 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354010040 con 0x7f435c0b01c0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.546+0000 7f4358ff9640 1 -- 172.21.15.183:0/1459798885 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4354014400 con 0x7f435c0b01c0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.546+0000 7f4358ff9640 1 -- 172.21.15.183:0/1459798885 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354009ea0 con 0x7f435c0b01c0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.546+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f434c009400 con 0x7f435c0b01c0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.546+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f435c1f7920 con 0x7f435c0b01c0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.546+0000 7f4358ff9640 1 -- 172.21.15.183:0/1459798885 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f43540147b0 con 0x7f435c0b01c0 2024-03-23T14:34:50.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.546+0000 7f4358ff9640 1 -- 172.21.15.183:0/1459798885 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f435401dbd0 con 0x7f435c0b01c0 2024-03-23T14:34:50.546 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.547+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f435c0736a0 con 0x7f435c0b01c0 2024-03-23T14:34:50.546 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.549+0000 7f4358ff9640 1 -- 172.21.15.183:0/1459798885 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f435403bdd0 con 0x7f435c0b01c0 2024-03-23T14:34:50.573 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.576+0000 7f819be73040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-23T14:34:50.587 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:50.591+0000 7f1bf5441040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T14:34:50.615 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.620+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a", "id": 4} v 0) -- 0x7f435c04f550 con 0x7f435c0b01c0 2024-03-23T14:34:50.619 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.624+0000 7f4358ff9640 1 -- 172.21.15.183:0/1459798885 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f435402a070 con 0x7f435c0b01c0 2024-03-23T14:34:50.619 INFO:teuthology.orchestra.run.smithi183.stdout:4 2024-03-23T14:34:50.620 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c0b01c0 msgr2=0x7f435c0cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 --2- 172.21.15.183:0/1459798885 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c0b01c0 0x7f435c0cfed0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f435400eed0 tx=0x7f435400b480 comp rx=0 tx=0).stop 2024-03-23T14:34:50.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 shutdown_connections 2024-03-23T14:34:50.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 --2- 172.21.15.183:0/1459798885 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f435c0b01c0 0x7f435c0cfed0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 --2- 172.21.15.183:0/1459798885 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f435c06b9a0 0x7f435c0cf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 --2- 172.21.15.183:0/1459798885 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f435c06b080 0x7f435c1f30e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 >> 172.21.15.183:0/1459798885 conn(0x7f435c0d0c60 msgr2=0x7f435c0b1220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 shutdown_connections 2024-03-23T14:34:50.621 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.625+0000 7f43618eb640 1 -- 172.21.15.183:0/1459798885 wait complete. 2024-03-23T14:34:50.627 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.630+0000 7f819be73040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-23T14:34:50.631 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 47e1e2a8-5abe-4727-9978-f52811fa76bd 5 2024-03-23T14:34:50.675 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:50.678+0000 7f819be73040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-23T14:34:50.705 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:50.709+0000 7f1bf5441040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T14:34:50.731 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.735+0000 7fa2657d7640 1 Processor -- start 2024-03-23T14:34:50.731 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.735+0000 7fa2657d7640 1 -- start start 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa2657d7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa260161750 0x7fa2600d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa2657d7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa260161b30 0x7fa2600d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa2657d7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2600d1330 con 0x7fa2600d1cb0 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa2657d7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa2600d0720 con 0x7fa260161750 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa2657d7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa2600d0a00 con 0x7fa260161b30 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa25f7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa260161b30 0x7fa2600d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa25f7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa260161b30 0x7fa2600d2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42582/0 (socket says 172.21.15.183:42582) 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa25f7fe640 1 -- 172.21.15.183:0/766123539 learned_addr learned my addr 172.21.15.183:0/766123539 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa25effd640 1 --2- 172.21.15.183:0/766123539 >> v2:172.21.15.161:3300/0 conn(0x7fa260161750 0x7fa2600d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa25f7fe640 1 -- 172.21.15.183:0/766123539 >> v2:172.21.15.161:3300/0 conn(0x7fa260161750 msgr2=0x7fa2600d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa25f7fe640 1 --2- 172.21.15.183:0/766123539 >> v2:172.21.15.161:3300/0 conn(0x7fa260161750 0x7fa2600d25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa25f7fe640 1 -- 172.21.15.183:0/766123539 >> v1:172.21.15.161:6789/0 conn(0x7fa2600d1cb0 legacy=0x7fa260162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.734 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.736+0000 7fa25f7fe640 1 -- 172.21.15.183:0/766123539 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2600d0ce0 con 0x7fa260161b30 2024-03-23T14:34:50.734 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.737+0000 7fa25f7fe640 1 --2- 172.21.15.183:0/766123539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa2600d2b00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa2600cb140 tx=0x7fa25802bf80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4374c3e6402cb9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.734 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.737+0000 7fa25dffb640 1 -- 172.21.15.183:0/766123539 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25802cb20 con 0x7fa260161b30 2024-03-23T14:34:50.734 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.737+0000 7fa25dffb640 1 -- 172.21.15.183:0/766123539 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25802cc80 con 0x7fa260161b30 2024-03-23T14:34:50.734 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.737+0000 7fa25dffb640 1 -- 172.21.15.183:0/766123539 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2580343f0 con 0x7fa260161b30 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.738+0000 7fa2657d7640 1 -- 172.21.15.183:0/766123539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 msgr2=0x7fa2600d2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.738+0000 7fa2657d7640 1 --2- 172.21.15.183:0/766123539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa2600d2b00 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa2600cb140 tx=0x7fa25802bf80 comp rx=0 tx=0).stop 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.738+0000 7fa2657d7640 1 -- 172.21.15.183:0/766123539 shutdown_connections 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.738+0000 7fa2657d7640 1 --2- 172.21.15.183:0/766123539 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa2600d2b00 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.738+0000 7fa2657d7640 1 --2- 172.21.15.183:0/766123539 >> v2:172.21.15.161:3300/0 conn(0x7fa260161750 0x7fa2600d25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.738+0000 7fa2657d7640 1 -- 172.21.15.183:0/766123539 >> 172.21.15.183:0/766123539 conn(0x7fa26009d450 msgr2=0x7fa2600d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.738+0000 7fa2657d7640 1 -- 172.21.15.183:0/766123539 shutdown_connections 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.738+0000 7fa2657d7640 1 -- 172.21.15.183:0/766123539 wait complete. 2024-03-23T14:34:50.735 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.739+0000 7fa2657d7640 1 Processor -- start 2024-03-23T14:34:50.736 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.739+0000 7fa2657d7640 1 -- start start 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa2657d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2600d1cb0 0x7fa260156af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa2657d7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa260161750 0x7fa2600c9910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa2657d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa26015aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa2657d7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa2600ca4f0 con 0x7fa260161750 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa2657d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa2601581b0 con 0x7fa260161b30 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa2657d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa2600ca210 con 0x7fa2600d1cb0 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25f7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa26015aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25effd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2600d1cb0 0x7fa260156af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25f7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa26015aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42598/0 (socket says 172.21.15.183:42598) 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25f7fe640 1 -- 172.21.15.183:0/4281356918 learned_addr learned my addr 172.21.15.183:0/4281356918 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.738 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25e7fc640 1 --2- 172.21.15.183:0/4281356918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa260161750 0x7fa2600c9910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25f7fe640 1 -- 172.21.15.183:0/4281356918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2600d1cb0 msgr2=0x7fa260156af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25f7fe640 1 --2- 172.21.15.183:0/4281356918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2600d1cb0 0x7fa260156af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25f7fe640 1 -- 172.21.15.183:0/4281356918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa260161750 msgr2=0x7fa2600c9910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25f7fe640 1 --2- 172.21.15.183:0/4281356918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa260161750 0x7fa2600c9910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.740+0000 7fa25f7fe640 1 -- 172.21.15.183:0/4281356918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa258009b30 con 0x7fa260161b30 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.741+0000 7fa25f7fe640 1 --2- 172.21.15.183:0/4281356918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa26015aba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa258036040 tx=0x7fa258004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.741+0000 7fa2477fe640 1 -- 172.21.15.183:0/4281356918 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa258009510 con 0x7fa260161b30 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.741+0000 7fa2477fe640 1 -- 172.21.15.183:0/4281356918 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa25802c920 con 0x7fa260161b30 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.741+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26015b8d0 con 0x7fa260161b30 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.741+0000 7fa2477fe640 1 -- 172.21.15.183:0/4281356918 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25803e670 con 0x7fa260161b30 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.741+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa26015be90 con 0x7fa260161b30 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.741+0000 7fa2477fe640 1 -- 172.21.15.183:0/4281356918 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa25803e7d0 con 0x7fa260161b30 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.741+0000 7fa2477fe640 1 -- 172.21.15.183:0/4281356918 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7fa258048de0 con 0x7fa260161b30 2024-03-23T14:34:50.739 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.742+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2600d3790 con 0x7fa260161b30 2024-03-23T14:34:50.740 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.744+0000 7fa2477fe640 1 -- 172.21.15.183:0/4281356918 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fa258066dd0 con 0x7fa260161b30 2024-03-23T14:34:50.803 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:50.807+0000 7f1bf5441040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T14:34:50.810 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.815+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "47e1e2a8-5abe-4727-9978-f52811fa76bd", "id": 5} v 0) -- 0x7fa26004f550 con 0x7fa260161b30 2024-03-23T14:34:50.813 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.818+0000 7fa2477fe640 1 -- 172.21.15.183:0/4281356918 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "47e1e2a8-5abe-4727-9978-f52811fa76bd", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7fa258055030 con 0x7fa260161b30 2024-03-23T14:34:50.813 INFO:teuthology.orchestra.run.smithi183.stdout:5 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 msgr2=0x7fa26015aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 --2- 172.21.15.183:0/4281356918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa26015aba0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fa258036040 tx=0x7fa258004a40 comp rx=0 tx=0).stop 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 shutdown_connections 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 --2- 172.21.15.183:0/4281356918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa260161b30 0x7fa26015aba0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 --2- 172.21.15.183:0/4281356918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa260161750 0x7fa2600c9910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 --2- 172.21.15.183:0/4281356918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa2600d1cb0 0x7fa260156af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 >> 172.21.15.183:0/4281356918 conn(0x7fa26009d450 msgr2=0x7fa26006bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 shutdown_connections 2024-03-23T14:34:50.815 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.819+0000 7fa2657d7640 1 -- 172.21.15.183:0/4281356918 wait complete. 2024-03-23T14:34:50.825 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new b65ec625-df08-474d-92af-bd7ab5d58510 6 2024-03-23T14:34:50.925 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.929+0000 7f2d530df640 1 Processor -- start 2024-03-23T14:34:50.925 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.929+0000 7f2d530df640 1 -- start start 2024-03-23T14:34:50.926 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d530df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d530df640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2d4c0d1cb0 0x7f2d4c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d530df640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d4c0d1330 con 0x7f2d4c161750 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d530df640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2d4c0d0720 con 0x7f2d4c0d1cb0 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d530df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4c0d0a00 con 0x7f2d4c0d2cd0 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d50e7b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2d4c161750 0x7f2d4c0d25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:47036/0 (socket says 172.21.15.183:47036) 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d50e7b640 1 -- 172.21.15.183:0/1379523445 learned_addr learned my addr 172.21.15.183:0/1379523445 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d4bfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d5167c640 1 --2- 172.21.15.183:0/1379523445 >> v2:172.21.15.161:3300/0 conn(0x7f2d4c0d1cb0 0x7f2d4c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d4b7fe640 1 -- 172.21.15.183:0/1379523445 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 763764152 0 0) 0x7f2d4c0d1330 con 0x7f2d4c161750 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d4bfff640 1 -- 172.21.15.183:0/1379523445 >> v2:172.21.15.161:3300/0 conn(0x7f2d4c0d1cb0 msgr2=0x7f2d4c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d4bfff640 1 --2- 172.21.15.183:0/1379523445 >> v2:172.21.15.161:3300/0 conn(0x7f2d4c0d1cb0 0x7f2d4c0cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d4bfff640 1 -- 172.21.15.183:0/1379523445 >> v1:172.21.15.183:6789/0 conn(0x7f2d4c161750 legacy=0x7f2d4c0d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.927 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.930+0000 7f2d4bfff640 1 -- 172.21.15.183:0/1379523445 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d4c0d0ce0 con 0x7f2d4c0d2cd0 2024-03-23T14:34:50.928 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.931+0000 7f2d4bfff640 1 --2- 172.21.15.183:0/1379523445 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0ceb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2d380097a0 tx=0x7f2d3802b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7f2c447669d63872 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.928 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.931+0000 7f2d4b7fe640 1 -- 172.21.15.183:0/1379523445 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38031830 con 0x7f2d4c0d2cd0 2024-03-23T14:34:50.928 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.931+0000 7f2d4b7fe640 1 -- 172.21.15.183:0/1379523445 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d38031990 con 0x7f2d4c0d2cd0 2024-03-23T14:34:50.928 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.931+0000 7f2d4b7fe640 1 -- 172.21.15.183:0/1379523445 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d38031d20 con 0x7f2d4c0d2cd0 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.932+0000 7f2d530df640 1 -- 172.21.15.183:0/1379523445 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d2cd0 msgr2=0x7f2d4c0ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.932+0000 7f2d530df640 1 --2- 172.21.15.183:0/1379523445 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0ceb10 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f2d380097a0 tx=0x7f2d3802b8d0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.932+0000 7f2d530df640 1 -- 172.21.15.183:0/1379523445 shutdown_connections 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.932+0000 7f2d530df640 1 --2- 172.21.15.183:0/1379523445 >> v2:172.21.15.161:3300/0 conn(0x7f2d4c0d1cb0 0x7f2d4c0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.932+0000 7f2d530df640 1 --2- 172.21.15.183:0/1379523445 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0ceb10 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.932+0000 7f2d530df640 1 -- 172.21.15.183:0/1379523445 >> 172.21.15.183:0/1379523445 conn(0x7f2d4c09d450 msgr2=0x7f2d4c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.932+0000 7f2d530df640 1 -- 172.21.15.183:0/1379523445 shutdown_connections 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.932+0000 7f2d530df640 1 -- 172.21.15.183:0/1379523445 wait complete. 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.933+0000 7f2d530df640 1 Processor -- start 2024-03-23T14:34:50.929 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.933+0000 7f2d530df640 1 -- start start 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d530df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d1cb0 0x7f2d4c1f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d530df640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0cb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d530df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d4c161750 0x7f2d4c0b2640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d530df640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c1f3b80 con 0x7f2d4c0d2cd0 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d530df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2d4c0cbbe0 con 0x7f2d4c161750 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d530df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2d4c0cbec0 con 0x7f2d4c0d1cb0 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d50e7b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d1cb0 0x7f2d4c1f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d4c161750 0x7f2d4c0b2640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.932 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d4c161750 0x7f2d4c0b2640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.183:42612/0 (socket says 172.21.15.183:42612) 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d50e7b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d1cb0 0x7f2d4c1f3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:38690/0 (socket says 172.21.15.183:38690) 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 -- 172.21.15.183:0/2865557731 learned_addr learned my addr 172.21.15.183:0/2865557731 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d4bfff640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0cb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 -- 172.21.15.183:0/2865557731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d1cb0 msgr2=0x7f2d4c1f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d1cb0 0x7f2d4c1f3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 -- 172.21.15.183:0/2865557731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d4c0d2cd0 msgr2=0x7f2d4c0cb1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0cb1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 -- 172.21.15.183:0/2865557731 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d38009400 con 0x7f2d4c161750 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d50e7b640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d1cb0 0x7f2d4c1f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.934+0000 7f2d5167c640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d4c161750 0x7f2d4c0b2640 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2d44007e40 tx=0x7f2d4400e4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.935+0000 7f2d497fa640 1 -- 172.21.15.183:0/2865557731 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d44014620 con 0x7f2d4c161750 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.935+0000 7f2d497fa640 1 -- 172.21.15.183:0/2865557731 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d4400ee90 con 0x7f2d4c161750 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.935+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d4c0cc1a0 con 0x7f2d4c161750 2024-03-23T14:34:50.933 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.935+0000 7f2d497fa640 1 -- 172.21.15.183:0/2865557731 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d440149e0 con 0x7f2d4c161750 2024-03-23T14:34:50.934 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.935+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d4c0b9850 con 0x7f2d4c161750 2024-03-23T14:34:50.934 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.935+0000 7f2d497fa640 1 -- 172.21.15.183:0/2865557731 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f2d44014b40 con 0x7f2d4c161750 2024-03-23T14:34:50.934 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.935+0000 7f2d497fa640 1 -- 172.21.15.183:0/2865557731 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f2d4401da00 con 0x7f2d4c161750 2024-03-23T14:34:50.934 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.936+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d4c0cc6c0 con 0x7f2d4c161750 2024-03-23T14:34:50.934 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:50.938+0000 7f2d497fa640 1 -- 172.21.15.183:0/2865557731 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f2d4c0cc6c0 con 0x7f2d4c161750 2024-03-23T14:34:50.966 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:50.970+0000 7f1bf5441040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T14:34:51.003 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.008+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd new", "uuid": "b65ec625-df08-474d-92af-bd7ab5d58510", "id": 6} v 0) -- 0x7f2d4c0b99e0 con 0x7f2d4c161750 2024-03-23T14:34:51.007 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.011+0000 7f2d497fa640 1 -- 172.21.15.183:0/2865557731 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b65ec625-df08-474d-92af-bd7ab5d58510", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f2d4402d0b0 con 0x7f2d4c161750 2024-03-23T14:34:51.007 INFO:teuthology.orchestra.run.smithi183.stdout:6 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d4c161750 msgr2=0x7f2d4c0b2640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d4c161750 0x7f2d4c0b2640 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f2d44007e40 tx=0x7f2d4400e4b0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 shutdown_connections 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d4c161750 0x7f2d4c0b2640 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d4c0d2cd0 0x7f2d4c0cb1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 --2- 172.21.15.183:0/2865557731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d4c0d1cb0 0x7f2d4c1f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 >> 172.21.15.183:0/2865557731 conn(0x7f2d4c09d450 msgr2=0x7f2d4c06d210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 shutdown_connections 2024-03-23T14:34:51.008 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.012+0000 7f2d530df640 1 -- 172.21.15.183:0/2865557731 wait complete. 2024-03-23T14:34:51.009 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:51.013+0000 7f1bf5441040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T14:34:51.018 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 0670b2b9-514f-4a9a-9da6-d35c981423dc 7 2024-03-23T14:34:51.056 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.060+0000 7f819be73040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-23T14:34:51.118 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.122+0000 7f885bafb640 1 Processor -- start 2024-03-23T14:34:51.118 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885bafb640 1 -- start start 2024-03-23T14:34:51.121 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885bafb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f885415ad80 0x7f885415f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.121 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885bafb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f8854071ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.121 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885bafb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f885415bd00 con 0x7f8854071b60 2024-03-23T14:34:51.121 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885bafb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8854072530 con 0x7f885415ad80 2024-03-23T14:34:51.121 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885bafb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f88540d0670 con 0x7f885415b870 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885a098640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f8854071ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885a098640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f8854071ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:38696/0 (socket says 172.21.15.183:38696) 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.123+0000 7f885a098640 1 -- 172.21.15.183:0/4162943734 learned_addr learned my addr 172.21.15.183:0/4162943734 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.124+0000 7f8859897640 1 --2- 172.21.15.183:0/4162943734 >> v2:172.21.15.161:3300/0 conn(0x7f885415ad80 0x7f885415f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.124+0000 7f885a098640 1 -- 172.21.15.183:0/4162943734 >> v2:172.21.15.161:3300/0 conn(0x7f885415ad80 msgr2=0x7f885415f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.124+0000 7f885a098640 1 --2- 172.21.15.183:0/4162943734 >> v2:172.21.15.161:3300/0 conn(0x7f885415ad80 0x7f885415f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.124+0000 7f885a098640 1 -- 172.21.15.183:0/4162943734 >> v1:172.21.15.161:6789/0 conn(0x7f8854071b60 legacy=0x7f885415b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.124+0000 7f885a098640 1 -- 172.21.15.183:0/4162943734 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f885415bf40 con 0x7f885415b870 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.124+0000 7f885a098640 1 --2- 172.21.15.183:0/4162943734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f8854071ff0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f884c009e50 tx=0x7f884c02be40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae007e710cdb1f56 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f8858895640 1 -- 172.21.15.183:0/4162943734 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c02c9e0 con 0x7f885415b870 2024-03-23T14:34:51.122 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f8858895640 1 -- 172.21.15.183:0/4162943734 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f884c02cb40 con 0x7f885415b870 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f8858895640 1 -- 172.21.15.183:0/4162943734 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c0343f0 con 0x7f885415b870 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f885bafb640 1 -- 172.21.15.183:0/4162943734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 msgr2=0x7f8854071ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f885bafb640 1 --2- 172.21.15.183:0/4162943734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f8854071ff0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f884c009e50 tx=0x7f884c02be40 comp rx=0 tx=0).stop 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f885bafb640 1 -- 172.21.15.183:0/4162943734 shutdown_connections 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f885bafb640 1 --2- 172.21.15.183:0/4162943734 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f8854071ff0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f885bafb640 1 --2- 172.21.15.183:0/4162943734 >> v2:172.21.15.161:3300/0 conn(0x7f885415ad80 0x7f885415f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f885bafb640 1 -- 172.21.15.183:0/4162943734 >> 172.21.15.183:0/4162943734 conn(0x7f88540d0c00 msgr2=0x7f88540d1010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f885bafb640 1 -- 172.21.15.183:0/4162943734 shutdown_connections 2024-03-23T14:34:51.123 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.125+0000 7f885bafb640 1 -- 172.21.15.183:0/4162943734 wait complete. 2024-03-23T14:34:51.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.126+0000 7f885bafb640 1 Processor -- start 2024-03-23T14:34:51.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.126+0000 7f885bafb640 1 -- start start 2024-03-23T14:34:51.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885bafb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8854071b60 0x7f88541f13d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885bafb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885415ad80 0x7f8854158110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885bafb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f88540c9ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885bafb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f88541f1910 con 0x7f885415ad80 2024-03-23T14:34:51.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885bafb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f88540b4840 con 0x7f8854071b60 2024-03-23T14:34:51.126 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885bafb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f88540b4a90 con 0x7f885415b870 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885a098640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f88540c9ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f8859897640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8854071b60 0x7f88541f13d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885a098640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f88540c9ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.183:38706/0 (socket says 172.21.15.183:38706) 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885a098640 1 -- 172.21.15.183:0/685432852 learned_addr learned my addr 172.21.15.183:0/685432852 (peer_addr_for_me v2:172.21.15.183:0/0) 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f8859096640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885415ad80 0x7f8854158110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885a098640 1 -- 172.21.15.183:0/685432852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8854071b60 msgr2=0x7f88541f13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885a098640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8854071b60 0x7f88541f13d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885a098640 1 -- 172.21.15.183:0/685432852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885415ad80 msgr2=0x7f8854158110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885a098640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885415ad80 0x7f8854158110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.127+0000 7f885a098640 1 -- 172.21.15.183:0/685432852 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f884c009b30 con 0x7f885415b870 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f8859897640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8854071b60 0x7f88541f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f8859096640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885415ad80 0x7f8854158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f885a098640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f88540c9ce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f884c009e50 tx=0x7f884c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:51.127 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f88467fc640 1 -- 172.21.15.183:0/685432852 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c007e50 con 0x7f885415b870 2024-03-23T14:34:51.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f88467fc640 1 -- 172.21.15.183:0/685432852 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f884c040070 con 0x7f885415b870 2024-03-23T14:34:51.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88540b4d70 con 0x7f885415b870 2024-03-23T14:34:51.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f88467fc640 1 -- 172.21.15.183:0/685432852 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f884c0476c0 con 0x7f885415b870 2024-03-23T14:34:51.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f88540b5330 con 0x7f885415b870 2024-03-23T14:34:51.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f88467fc640 1 -- 172.21.15.183:0/685432852 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f884c047bb0 con 0x7f885415b870 2024-03-23T14:34:51.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.128+0000 7f88467fc640 1 -- 172.21.15.183:0/685432852 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f884c042070 con 0x7f885415b870 2024-03-23T14:34:51.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.129+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8854073510 con 0x7f885415b870 2024-03-23T14:34:51.128 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.131+0000 7f88467fc640 1 -- 172.21.15.183:0/685432852 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f884c03cd60 con 0x7f885415b870 2024-03-23T14:34:51.171 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:51.176+0000 7f1bf5441040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T14:34:51.190 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.194+0000 7f819be73040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-23T14:34:51.198 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.203+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd new", "uuid": "0670b2b9-514f-4a9a-9da6-d35c981423dc", "id": 7} v 0) -- 0x7f885404f550 con 0x7f885415b870 2024-03-23T14:34:51.202 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.207+0000 7f88467fc640 1 -- 172.21.15.183:0/685432852 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0670b2b9-514f-4a9a-9da6-d35c981423dc", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f884c03c450 con 0x7f885415b870 2024-03-23T14:34:51.202 INFO:teuthology.orchestra.run.smithi183.stdout:7 2024-03-23T14:34:51.203 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.207+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 msgr2=0x7f88540c9ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.203 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.207+0000 7f885bafb640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f88540c9ce0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f884c009e50 tx=0x7f884c004a40 comp rx=0 tx=0).stop 2024-03-23T14:34:51.203 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.207+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 shutdown_connections 2024-03-23T14:34:51.203 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.207+0000 7f885bafb640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f885415b870 0x7f88540c9ce0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.203 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.207+0000 7f885bafb640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f885415ad80 0x7f8854158110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.207+0000 7f885bafb640 1 --2- 172.21.15.183:0/685432852 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8854071b60 0x7f88541f13d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.207+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 >> 172.21.15.183:0/685432852 conn(0x7f88540d0c00 msgr2=0x7f8854159820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:51.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.208+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 shutdown_connections 2024-03-23T14:34:51.204 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:34:51.208+0000 7f885bafb640 1 -- 172.21.15.183:0/685432852 wait complete. 2024-03-23T14:34:51.214 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-23T14:34:51.214 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-23T14:34:51.216 INFO:tasks.ceph.osd.0:Started 2024-03-23T14:34:51.216 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-23T14:34:51.216 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-23T14:34:51.219 INFO:tasks.ceph.osd.1:Started 2024-03-23T14:34:51.219 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-23T14:34:51.219 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-23T14:34:51.221 INFO:tasks.ceph.osd.2:Started 2024-03-23T14:34:51.221 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-23T14:34:51.221 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-23T14:34:51.223 INFO:tasks.ceph.osd.3:Started 2024-03-23T14:34:51.223 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-23T14:34:51.223 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-23T14:34:51.226 INFO:tasks.ceph.osd.4:Started 2024-03-23T14:34:51.226 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-23T14:34:51.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-23T14:34:51.228 INFO:tasks.ceph.osd.5:Started 2024-03-23T14:34:51.228 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-23T14:34:51.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-23T14:34:51.230 INFO:tasks.ceph.osd.6:Started 2024-03-23T14:34:51.230 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-23T14:34:51.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-23T14:34:51.234 INFO:tasks.ceph.osd.7:Started 2024-03-23T14:34:51.235 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:34:51.346 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.350+0000 7f819be73040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-23T14:34:51.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.359+0000 7fc7639d0640 1 Processor -- start 2024-03-23T14:34:51.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.360+0000 7fc7639d0640 1 -- start start 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.360+0000 7fc7639d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c0d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.360+0000 7fc7639d0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc75c0d2970 0x7fc75c0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.360+0000 7fc7639d0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc75c0d2dd0 con 0x7fc75c15e400 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc7639d0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc75c0d1050 con 0x7fc75c0d2970 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc7639d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc75c0d0460 con 0x7fc75c0d1470 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc761f6d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc75c0d2970 0x7fc75c0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc76176c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc75c15e400 0x7fc75c0cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:53338/0 (socket says 172.21.15.161:53338) 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc76176c640 1 -- 172.21.15.161:0/1463084785 learned_addr learned my addr 172.21.15.161:0/1463084785 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:34:51.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc760f6b640 1 --2- 172.21.15.161:0/1463084785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c0d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc761f6d640 1 -- 172.21.15.161:0/1463084785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 msgr2=0x7fc75c0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:34:51.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc761f6d640 1 --2- 172.21.15.161:0/1463084785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c0d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc761f6d640 1 -- 172.21.15.161:0/1463084785 >> v1:172.21.15.183:6789/0 conn(0x7fc75c15e400 legacy=0x7fc75c0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.361+0000 7fc761f6d640 1 -- 172.21.15.161:0/1463084785 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc75c0d06b0 con 0x7fc75c0d2970 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.362+0000 7fc761f6d640 1 --2- 172.21.15.161:0/1463084785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 0x7fc75c0b0a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc754009d90 tx=0x7fc75402beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=809f993e598f5566 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.362+0000 7fc753fff640 1 -- 172.21.15.161:0/1463084785 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75402cc60 con 0x7fc75c0d2970 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.362+0000 7fc753fff640 1 -- 172.21.15.161:0/1463084785 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75402cdc0 con 0x7fc75c0d2970 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.362+0000 7fc753fff640 1 -- 172.21.15.161:0/1463084785 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc754034660 con 0x7fc75c0d2970 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.362+0000 7fc7639d0640 1 -- 172.21.15.161:0/1463084785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 msgr2=0x7fc75c0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.362+0000 7fc7639d0640 1 --2- 172.21.15.161:0/1463084785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 0x7fc75c0b0a80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fc754009d90 tx=0x7fc75402beb0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.363+0000 7fc7639d0640 1 -- 172.21.15.161:0/1463084785 shutdown_connections 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.363+0000 7fc7639d0640 1 --2- 172.21.15.161:0/1463084785 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 0x7fc75c0b0a80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.363+0000 7fc7639d0640 1 --2- 172.21.15.161:0/1463084785 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c0d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.363+0000 7fc7639d0640 1 -- 172.21.15.161:0/1463084785 >> 172.21.15.161:0/1463084785 conn(0x7fc75c09d190 msgr2=0x7fc75c0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:51.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.363+0000 7fc7639d0640 1 -- 172.21.15.161:0/1463084785 shutdown_connections 2024-03-23T14:34:51.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.363+0000 7fc7639d0640 1 -- 172.21.15.161:0/1463084785 wait complete. 2024-03-23T14:34:51.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.364+0000 7fc7639d0640 1 Processor -- start 2024-03-23T14:34:51.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.364+0000 7fc7639d0640 1 -- start start 2024-03-23T14:34:51.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc7639d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c15ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc7639d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 0x7fc75c0c8590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc7639d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc75c15e400 0x7fc75c0cd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:51.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc7639d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c0cdde0 con 0x7fc75c15e400 2024-03-23T14:34:51.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc7639d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc75c0c8ad0 con 0x7fc75c0d2970 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc7639d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc75c0cdb00 con 0x7fc75c0d1470 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c15ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc761f6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc75c15e400 0x7fc75c0cd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c15ca50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:56316/0 (socket says 172.21.15.161:56316) 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc760f6b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 0x7fc75c0c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 -- 172.21.15.161:0/170378493 learned_addr learned my addr 172.21.15.161:0/170378493 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 -- 172.21.15.161:0/170378493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 msgr2=0x7fc75c0c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 --2- 172.21.15.161:0/170378493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 0x7fc75c0c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 -- 172.21.15.161:0/170378493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc75c15e400 msgr2=0x7fc75c0cd0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 --2- 172.21.15.161:0/170378493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc75c15e400 0x7fc75c0cd0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 -- 172.21.15.161:0/170378493 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc754009a70 con 0x7fc75c0d1470 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.365+0000 7fc76176c640 1 --2- 172.21.15.161:0/170378493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c15ca50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00c480 tx=0x7fc74c00c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.366+0000 7fc751ffb640 1 -- 172.21.15.161:0/170378493 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c005ce0 con 0x7fc75c0d1470 2024-03-23T14:34:51.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.366+0000 7fc751ffb640 1 -- 172.21.15.161:0/170378493 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc74c005e40 con 0x7fc75c0d1470 2024-03-23T14:34:51.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.366+0000 7fc751ffb640 1 -- 172.21.15.161:0/170378493 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc74c0072d0 con 0x7fc75c0d1470 2024-03-23T14:34:51.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.366+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc75c0c7ab0 con 0x7fc75c0d1470 2024-03-23T14:34:51.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.366+0000 7fc751ffb640 1 -- 172.21.15.161:0/170378493 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc74c0074f0 con 0x7fc75c0d1470 2024-03-23T14:34:51.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.366+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc75c0c8070 con 0x7fc75c0d1470 2024-03-23T14:34:51.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.366+0000 7fc751ffb640 1 -- 172.21.15.161:0/170378493 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fc74c01ca80 con 0x7fc75c0d1470 2024-03-23T14:34:51.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.367+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc75c0a1e00 con 0x7fc75c0d1470 2024-03-23T14:34:51.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.369+0000 7fc751ffb640 1 -- 172.21.15.161:0/170378493 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fc74c018070 con 0x7fc75c0d1470 2024-03-23T14:34:51.394 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.399+0000 7f819be73040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-23T14:34:51.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.430+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc75c0c8260 con 0x7fc75c0d1470 2024-03-23T14:34:51.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.431+0000 7fc751ffb640 1 -- 172.21.15.161:0/170378493 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fc74c01c3b0 con 0x7fc75c0d1470 2024-03-23T14:34:51.426 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:34:51.427 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":10,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:34:51.204801+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 msgr2=0x7fc75c15ca50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 --2- 172.21.15.161:0/170378493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c15ca50 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc74c00c480 tx=0x7fc74c00c940 comp rx=0 tx=0).stop 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 shutdown_connections 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 --2- 172.21.15.161:0/170378493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc75c15e400 0x7fc75c0cd0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 --2- 172.21.15.161:0/170378493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc75c0d2970 0x7fc75c0c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 --2- 172.21.15.161:0/170378493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc75c0d1470 0x7fc75c15ca50 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 >> 172.21.15.161:0/170378493 conn(0x7fc75c09d190 msgr2=0x7fc75c160bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 shutdown_connections 2024-03-23T14:34:51.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:34:51.432+0000 7fc7639d0640 1 -- 172.21.15.161:0/170378493 wait complete. 2024-03-23T14:34:51.438 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-23T14:34:51.438 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-23T14:34:51.511 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.515+0000 7f819be73040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-23T14:34:51.609 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.613+0000 7f819be73040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-23T14:34:51.761 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.765+0000 7f819be73040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-23T14:34:51.804 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.809+0000 7f819be73040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-23T14:34:51.832 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-03-23T14:34:51.837+0000 7f93f5c11640 -1 Falling back to public interface 2024-03-23T14:34:51.840 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-03-23T14:34:51.845+0000 7fedb8116640 -1 Falling back to public interface 2024-03-23T14:34:51.841 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-23T14:34:51.845+0000 7fa737b73640 -1 Falling back to public interface 2024-03-23T14:34:51.841 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-23T14:34:51.845+0000 7fa736b22640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6808/0: (98) Address already in use 2024-03-23T14:34:51.842 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-03-23T14:34:51.846+0000 7f8f62e11640 -1 Falling back to public interface 2024-03-23T14:34:51.845 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-23T14:34:51.849+0000 7f31a6666640 -1 Falling back to public interface 2024-03-23T14:34:51.847 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-23T14:34:51.851+0000 7fcd4f80d640 -1 Falling back to public interface 2024-03-23T14:34:51.854 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-23T14:34:51.858+0000 7f8e79d73640 -1 Falling back to public interface 2024-03-23T14:34:51.855 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-23T14:34:51.859+0000 7fd8d0ace640 -1 Falling back to public interface 2024-03-23T14:34:51.964 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:51.968+0000 7f819be73040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-23T14:34:52.002 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:52.007+0000 7f1bf5441040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T14:34:52.244 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T14:34:52.279 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:52.283+0000 7f1bf5441040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T14:34:52.330 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:52.334+0000 7f1bf5441040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T14:34:52.335 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.340+0000 7f6c06fa5640 1 Processor -- start 2024-03-23T14:34:52.336 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.340+0000 7f6c06fa5640 1 -- start start 2024-03-23T14:34:52.337 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6c06fa5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6c000d20f0 0x7f6c0015fb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:52.337 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6c06fa5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c00160a90 con 0x7f6c000ce1b0 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6c06fa5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6c000b1810 con 0x7f6c000cddd0 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6c06fa5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f6c000d0320 con 0x7f6c000d20f0 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6bfffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6c000ce1b0 0x7f6c000d19e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53954/0 (socket says 172.21.15.161:53954) 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6bfffff640 1 -- 172.21.15.161:0/1768312156 learned_addr learned my addr 172.21.15.161:0/1768312156 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6c05542640 1 --2- 172.21.15.161:0/1768312156 >> v2:172.21.15.183:3300/0 conn(0x7f6c000d20f0 0x7f6c0015fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6bff7fe640 1 -- 172.21.15.161:0/1768312156 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2249083940 0 0) 0x7f6c00160a90 con 0x7f6c000ce1b0 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6bff7fe640 1 -- 172.21.15.161:0/1768312156 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6bf0003520 con 0x7f6c000ce1b0 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6bff7fe640 1 -- 172.21.15.161:0/1768312156 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2747421156 0 0) 0x7f6c000b1810 con 0x7f6c000cddd0 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6bff7fe640 1 -- 172.21.15.161:0/1768312156 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c00160a90 con 0x7f6c000cddd0 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6c05542640 1 -- 172.21.15.161:0/1768312156 >> v1:172.21.15.183:6789/0 conn(0x7f6c000cddd0 legacy=0x7f6c000cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6c05542640 1 -- 172.21.15.161:0/1768312156 >> v1:172.21.15.161:6789/0 conn(0x7f6c000ce1b0 legacy=0x7f6c000d19e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:52.338 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.341+0000 7f6c05542640 1 -- 172.21.15.161:0/1768312156 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c001601f0 con 0x7f6c000d20f0 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.342+0000 7f6c05542640 1 --2- 172.21.15.161:0/1768312156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000d20f0 0x7f6c0015fb80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8009d90 tx=0x7f6bf802beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5282c10ac295c96a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6bff7fe640 1 -- 172.21.15.161:0/1768312156 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf802cc60 con 0x7f6c000d20f0 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6bff7fe640 1 -- 172.21.15.161:0/1768312156 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf802cdc0 con 0x7f6c000d20f0 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6bff7fe640 1 -- 172.21.15.161:0/1768312156 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf8034660 con 0x7f6c000d20f0 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6c06fa5640 1 -- 172.21.15.161:0/1768312156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000d20f0 msgr2=0x7f6c0015fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6c06fa5640 1 --2- 172.21.15.161:0/1768312156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000d20f0 0x7f6c0015fb80 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6bf8009d90 tx=0x7f6bf802beb0 comp rx=0 tx=0).stop 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6c06fa5640 1 -- 172.21.15.161:0/1768312156 shutdown_connections 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6c06fa5640 1 --2- 172.21.15.161:0/1768312156 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000d20f0 0x7f6c0015fb80 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6c06fa5640 1 -- 172.21.15.161:0/1768312156 >> 172.21.15.161:0/1768312156 conn(0x7f6c0009a9f0 msgr2=0x7f6c0009ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6c06fa5640 1 -- 172.21.15.161:0/1768312156 shutdown_connections 2024-03-23T14:34:52.340 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.343+0000 7f6c06fa5640 1 -- 172.21.15.161:0/1768312156 wait complete. 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.345+0000 7f6c06fa5640 1 Processor -- start 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.345+0000 7f6c06fa5640 1 -- start start 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.345+0000 7f6c06fa5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000cddd0 0x7f6c000b75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.345+0000 7f6c06fa5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c000ce1b0 0x7f6c000c6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.345+0000 7f6c06fa5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c000d20f0 0x7f6c00159700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.345+0000 7f6c06fa5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6c000c6cd0 con 0x7f6c000d20f0 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.345+0000 7f6c06fa5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6c000c7b10 con 0x7f6c000cddd0 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.345+0000 7f6c06fa5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f6c000c6f00 con 0x7f6c000ce1b0 2024-03-23T14:34:52.343 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c05542640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c000d20f0 0x7f6c00159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000cddd0 0x7f6c000b75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000cddd0 0x7f6c000b75d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:43488/0 (socket says 172.21.15.161:43488) 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c05542640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c000d20f0 0x7f6c00159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58204/0 (socket says 172.21.15.161:58204) 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6bfffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c000ce1b0 0x7f6c000c6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 -- 172.21.15.161:0/3907820316 learned_addr learned my addr 172.21.15.161:0/3907820316 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 -- 172.21.15.161:0/3907820316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c000ce1b0 msgr2=0x7f6c000c6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 --2- 172.21.15.161:0/3907820316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c000ce1b0 0x7f6c000c6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 -- 172.21.15.161:0/3907820316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c000d20f0 msgr2=0x7f6c00159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 --2- 172.21.15.161:0/3907820316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c000d20f0 0x7f6c00159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 -- 172.21.15.161:0/3907820316 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf8009a70 con 0x7f6c000cddd0 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c05542640 1 --2- 172.21.15.161:0/3907820316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c000d20f0 0x7f6c00159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:34:52.344 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c04d41640 1 --2- 172.21.15.161:0/3907820316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000cddd0 0x7f6c000b75d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400c500 tx=0x7f6bf400c9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6bfd7fa640 1 -- 172.21.15.161:0/3907820316 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf40052d0 con 0x7f6c000cddd0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6bfd7fa640 1 -- 172.21.15.161:0/3907820316 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6bf4005450 con 0x7f6c000cddd0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c000c71e0 con 0x7f6c000cddd0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6bfd7fa640 1 -- 172.21.15.161:0/3907820316 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6bf4012870 con 0x7f6c000cddd0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.346+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c000c77a0 con 0x7f6c000cddd0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.347+0000 7f6bfd7fa640 1 -- 172.21.15.161:0/3907820316 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f6bf40129d0 con 0x7f6c000cddd0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.347+0000 7f6bfd7fa640 1 -- 172.21.15.161:0/3907820316 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f6bf4012c80 con 0x7f6c000cddd0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.347+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c000c7930 con 0x7f6c000cddd0 2024-03-23T14:34:52.345 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.349+0000 7f6bfd7fa640 1 -- 172.21.15.161:0/3907820316 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f6bf40035f0 con 0x7f6c000cddd0 2024-03-23T14:34:52.407 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.411+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c0004f470 con 0x7f6c000cddd0 2024-03-23T14:34:52.407 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.412+0000 7f6bfd7fa640 1 -- 172.21.15.161:0/3907820316 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f6bf4003790 con 0x7f6c000cddd0 2024-03-23T14:34:52.407 INFO:teuthology.misc.health.smithi161.stdout: 2024-03-23T14:34:52.408 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":10,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:34:51.204801+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000cddd0 msgr2=0x7f6c000b75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 --2- 172.21.15.161:0/3907820316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000cddd0 0x7f6c000b75d0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6bf400c500 tx=0x7f6bf400c9c0 comp rx=0 tx=0).stop 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 shutdown_connections 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 --2- 172.21.15.161:0/3907820316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6c000d20f0 0x7f6c00159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 --2- 172.21.15.161:0/3907820316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6c000ce1b0 0x7f6c000c6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 --2- 172.21.15.161:0/3907820316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6c000cddd0 0x7f6c000b75d0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 >> 172.21.15.161:0/3907820316 conn(0x7f6c0009a9f0 msgr2=0x7f6c000cc820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 shutdown_connections 2024-03-23T14:34:52.409 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:52.413+0000 7f6c06fa5640 1 -- 172.21.15.161:0/3907820316 wait complete. 2024-03-23T14:34:52.418 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-23T14:34:52.447 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:52.451+0000 7f1bf5441040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T14:34:52.785 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:52.789+0000 7f819be73040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-23T14:34:53.066 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:53.070+0000 7f819be73040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-23T14:34:53.116 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:53.121+0000 7f819be73040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-23T14:34:53.232 INFO:tasks.ceph.mgr.y.smithi161.stderr:2024-03-23T14:34:53.237+0000 7f819be73040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-23T14:34:53.474 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:53.478+0000 7f1b96108640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-23T14:34:53.474 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-23T14:34:53.479+0000 7f1b96108640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-23T14:34:55.297 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-03-23T14:34:55.301+0000 7f93f5c11640 -1 osd.3 0 log_to_monitors true 2024-03-23T14:34:55.305 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-23T14:34:55.309+0000 7f31a6666640 -1 osd.6 0 log_to_monitors true 2024-03-23T14:34:55.379 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-23T14:34:55.382+0000 7fcd4f80d640 -1 osd.7 0 log_to_monitors true 2024-03-23T14:34:55.379 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-23T14:34:55.382+0000 7f8e79d73640 -1 osd.5 0 log_to_monitors true 2024-03-23T14:34:55.380 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-03-23T14:34:55.384+0000 7fedb8116640 -1 osd.1 0 log_to_monitors true 2024-03-23T14:34:55.384 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-23T14:34:55.387+0000 7fd8d0ace640 -1 osd.4 0 log_to_monitors true 2024-03-23T14:34:55.391 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-23T14:34:55.396+0000 7fa737b73640 -1 osd.2 0 log_to_monitors true 2024-03-23T14:34:55.395 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-03-23T14:34:55.399+0000 7f8f62e11640 -1 osd.0 0 log_to_monitors true 2024-03-23T14:34:58.567 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-23T14:34:58.570+0000 7fa7332e6640 -1 osd.2 0 waiting for initial osdmap 2024-03-23T14:34:58.571 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-03-23T14:34:58.575+0000 7f93f0e52640 -1 osd.3 0 waiting for initial osdmap 2024-03-23T14:34:58.576 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-23T14:34:58.580+0000 7fcd4c16e640 -1 osd.7 0 waiting for initial osdmap 2024-03-23T14:34:58.577 INFO:tasks.ceph.osd.2.smithi161.stderr:2024-03-23T14:34:58.582+0000 7fa72b918640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T14:34:58.578 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-23T14:34:58.581+0000 7fd8cc241640 -1 osd.4 0 waiting for initial osdmap 2024-03-23T14:34:58.580 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-23T14:34:58.584+0000 7f8e754e6640 -1 osd.5 0 waiting for initial osdmap 2024-03-23T14:34:58.581 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-23T14:34:58.585+0000 7f31a3611640 -1 osd.6 0 waiting for initial osdmap 2024-03-23T14:34:58.585 INFO:tasks.ceph.osd.3.smithi161.stderr:2024-03-23T14:34:58.590+0000 7f93e9484640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T14:34:58.590 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-23T14:34:58.593+0000 7fd8c4873640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T14:34:58.590 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-23T14:34:58.593+0000 7fcd42f68640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T14:34:58.598 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-23T14:34:58.602+0000 7f319a40b640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T14:34:58.599 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-23T14:34:58.603+0000 7f8e6db18640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T14:34:58.602 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-03-23T14:34:58.606+0000 7fedb3889640 -1 osd.1 0 waiting for initial osdmap 2024-03-23T14:34:58.612 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-03-23T14:34:58.617+0000 7f8f5dfb0640 -1 osd.0 0 waiting for initial osdmap 2024-03-23T14:34:58.613 INFO:tasks.ceph.osd.1.smithi161.stderr:2024-03-23T14:34:58.618+0000 7fedabebb640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T14:34:58.633 INFO:tasks.ceph.osd.0.smithi161.stderr:2024-03-23T14:34:58.638+0000 7f8f565e2640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-23T14:34:59.225 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T14:34:59.319 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.323+0000 7febff9c7640 1 Processor -- start 2024-03-23T14:34:59.319 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.324+0000 7febff9c7640 1 -- start start 2024-03-23T14:34:59.321 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.324+0000 7febff9c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf80d1fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:59.321 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.324+0000 7febff9c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febf80d0560 con 0x7febf80d2db0 2024-03-23T14:34:59.321 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.324+0000 7febff9c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7febf80b0d20 con 0x7febf80cf720 2024-03-23T14:34:59.321 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.324+0000 7febff9c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7febf80b3360 con 0x7febf815e460 2024-03-23T14:34:59.321 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febfdf64640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7febf80d2db0 0x7febf80b11b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:53964/0 (socket says 172.21.15.161:53964) 2024-03-23T14:34:59.321 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febfdf64640 1 -- 172.21.15.161:0/35508478 learned_addr learned my addr 172.21.15.161:0/35508478 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:34:59.321 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febfd763640 1 --2- 172.21.15.161:0/35508478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf80d1fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:59.321 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febeffff640 1 -- 172.21.15.161:0/35508478 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1107070331 0 0) 0x7febf80d0560 con 0x7febf80d2db0 2024-03-23T14:34:59.322 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febeffff640 1 -- 172.21.15.161:0/35508478 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febe4003520 con 0x7febf80d2db0 2024-03-23T14:34:59.322 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febeffff640 1 -- 172.21.15.161:0/35508478 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789706875 0 0) 0x7febf80b0d20 con 0x7febf80cf720 2024-03-23T14:34:59.322 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febeffff640 1 -- 172.21.15.161:0/35508478 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febf80d0560 con 0x7febf80cf720 2024-03-23T14:34:59.322 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febeffff640 1 -- 172.21.15.161:0/35508478 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3004858361 0 0) 0x7febe4003520 con 0x7febf80d2db0 2024-03-23T14:34:59.322 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febfd763640 1 -- 172.21.15.161:0/35508478 >> v1:172.21.15.183:6789/0 conn(0x7febf80cf720 legacy=0x7febf80d26a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:59.322 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febfd763640 1 -- 172.21.15.161:0/35508478 >> v1:172.21.15.161:6789/0 conn(0x7febf80d2db0 legacy=0x7febf80b11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:59.322 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.325+0000 7febfd763640 1 -- 172.21.15.161:0/35508478 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febf80cfcc0 con 0x7febf815e460 2024-03-23T14:34:59.323 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.326+0000 7febfd763640 1 --2- 172.21.15.161:0/35508478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf80d1fc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7febe80097d0 tx=0x7febe802b770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae6a563cd4f5ebf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:59.323 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.326+0000 7febeffff640 1 -- 172.21.15.161:0/35508478 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf00019f0 con 0x7febf815e460 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.326+0000 7febeffff640 1 -- 172.21.15.161:0/35508478 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febe802e050 con 0x7febf815e460 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.326+0000 7febeffff640 1 -- 172.21.15.161:0/35508478 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febe8030730 con 0x7febf815e460 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.327+0000 7febff9c7640 1 -- 172.21.15.161:0/35508478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 msgr2=0x7febf80d1fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.327+0000 7febff9c7640 1 --2- 172.21.15.161:0/35508478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf80d1fc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7febe80097d0 tx=0x7febe802b770 comp rx=0 tx=0).stop 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.327+0000 7febff9c7640 1 -- 172.21.15.161:0/35508478 shutdown_connections 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.327+0000 7febff9c7640 1 --2- 172.21.15.161:0/35508478 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf80d1fc0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.327+0000 7febff9c7640 1 -- 172.21.15.161:0/35508478 >> 172.21.15.161:0/35508478 conn(0x7febf80d3eb0 msgr2=0x7febf80d42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.327+0000 7febff9c7640 1 -- 172.21.15.161:0/35508478 shutdown_connections 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.327+0000 7febff9c7640 1 -- 172.21.15.161:0/35508478 wait complete. 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.328+0000 7febff9c7640 1 Processor -- start 2024-03-23T14:34:59.324 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.328+0000 7febff9c7640 1 -- start start 2024-03-23T14:34:59.326 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febff9c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febf80cf720 0x7febf815eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:59.326 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febff9c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febf80d2db0 0x7febf80cc030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febff9c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf806cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febff9c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7febf80cddb0 con 0x7febf80d2db0 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febff9c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7febf80cc570 con 0x7febf80cf720 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febff9c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7febf815f150 con 0x7febf815e460 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febf80d2db0 0x7febf80cc030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febf80d2db0 0x7febf80cc030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:58250/0 (socket says 172.21.15.161:58250) 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 -- 172.21.15.161:0/1444882644 learned_addr learned my addr 172.21.15.161:0/1444882644 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfdf64640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf806cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfd763640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febf80cf720 0x7febf815eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 -- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 msgr2=0x7febf806cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf806cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 -- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febf80cf720 msgr2=0x7febf815eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febf80cf720 0x7febf815eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:59.327 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 -- 172.21.15.161:0/1444882644 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febe8009400 con 0x7febf80d2db0 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfd763640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febf80cf720 0x7febf815eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febfcf62640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febf80d2db0 0x7febf80cc030 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febf400c430 tx=0x7febf400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febedffb640 1 -- 172.21.15.161:0/1444882644 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf40052d0 con 0x7febf80d2db0 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febedffb640 1 -- 172.21.15.161:0/1444882644 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7febf4005450 con 0x7febf80d2db0 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febf815f430 con 0x7febf80d2db0 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.329+0000 7febedffb640 1 -- 172.21.15.161:0/1444882644 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febf4003830 con 0x7febf80d2db0 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.330+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febf815f9f0 con 0x7febf80d2db0 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.331+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febf804f470 con 0x7febf80d2db0 2024-03-23T14:34:59.328 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.331+0000 7febedffb640 1 -- 172.21.15.161:0/1444882644 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 4) ==== 89724+0+0 (secure 0 0 0) 0x7febf4003990 con 0x7febf80d2db0 2024-03-23T14:34:59.329 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.331+0000 7febedffb640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7febdc06b780 0x7febdc06dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:34:59.329 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.331+0000 7febedffb640 1 -- 172.21.15.161:0/1444882644 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(12..12 src has 1..12) ==== 2231+0+0 (secure 0 0 0) 0x7febf4088810 con 0x7febf80d2db0 2024-03-23T14:34:59.329 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.331+0000 7febfd763640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7febdc06b780 0x7febdc06dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:34:59.329 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.332+0000 7febfd763640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7febdc06b780 0x7febdc06dc40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7febe8034000 tx=0x7febe8008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:34:59.331 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.336+0000 7febedffb640 1 -- 172.21.15.161:0/1444882644 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7febf40541e0 con 0x7febf80d2db0 2024-03-23T14:34:59.508 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.512+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febf80b1a80 con 0x7febf80d2db0 2024-03-23T14:34:59.508 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.512+0000 7febedffb640 1 -- 172.21.15.161:0/1444882644 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v12) ==== 74+0+7058 (secure 0 0 0) 0x7febf4017090 con 0x7febf80d2db0 2024-03-23T14:34:59.508 INFO:teuthology.misc.health.smithi161.stdout: 2024-03-23T14:34:59.508 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":12,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:34:56.479899+0000","last_up_change":"0.000000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7febdc06b780 msgr2=0x7febdc06dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7febdc06b780 0x7febdc06dc40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7febe8034000 tx=0x7febe8008030 comp rx=0 tx=0).stop 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febf80d2db0 msgr2=0x7febf80cc030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febf80d2db0 0x7febf80cc030 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7febf400c430 tx=0x7febf400c8f0 comp rx=0 tx=0).stop 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 shutdown_connections 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7febdc06b780 0x7febdc06dc40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7febf815e460 0x7febf806cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7febf80d2db0 0x7febf80cc030 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 --2- 172.21.15.161:0/1444882644 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7febf80cf720 0x7febf815eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:34:59.511 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 >> 172.21.15.161:0/1444882644 conn(0x7febf80d3eb0 msgr2=0x7febf80d4290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:34:59.512 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 shutdown_connections 2024-03-23T14:34:59.512 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:34:59.515+0000 7febff9c7640 1 -- 172.21.15.161:0/1444882644 wait complete. 2024-03-23T14:34:59.520 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-23T14:35:06.327 DEBUG:teuthology.orchestra.run.smithi161:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:06.427 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.431+0000 7f22cb102640 1 Processor -- start 2024-03-23T14:35:06.428 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.432+0000 7f22cb102640 1 -- start start 2024-03-23T14:35:06.430 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.432+0000 7f22cb102640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22c406e620 0x7f22c4070e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.430 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.432+0000 7f22cb102640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.432+0000 7f22cb102640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22c40b3d30 con 0x7f22c406ea00 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.432+0000 7f22cb102640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f22c40ce1c0 con 0x7f22c406e620 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.432+0000 7f22cb102640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f22c40714c0 con 0x7f22c4072560 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.432+0000 7f22c8e9e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22c406e620 0x7f22c4070e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c8e9e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22c406e620 0x7f22c4070e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42448/0 (socket says 172.21.15.161:42448) 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c8e9e640 1 -- 172.21.15.161:0/2034709058 learned_addr learned my addr 172.21.15.161:0/2034709058 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c3fff640 1 --2- 172.21.15.161:0/2034709058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c8e9e640 1 -- 172.21.15.161:0/2034709058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 msgr2=0x7f22c40b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:06.431 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c8e9e640 1 --2- 172.21.15.161:0/2034709058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c8e9e640 1 -- 172.21.15.161:0/2034709058 >> v1:172.21.15.183:6789/0 conn(0x7f22c406ea00 legacy=0x7f22c415e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c8e9e640 1 -- 172.21.15.161:0/2034709058 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22c40717a0 con 0x7f22c406e620 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c3fff640 1 --2- 172.21.15.161:0/2034709058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.433+0000 7f22c8e9e640 1 --2- 172.21.15.161:0/2034709058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406e620 0x7f22c4070e30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009870 tx=0x7f22b802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac6e6fe95b33940f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22c37fe640 1 -- 172.21.15.161:0/2034709058 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8031640 con 0x7f22c406e620 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22c37fe640 1 -- 172.21.15.161:0/2034709058 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b80317a0 con 0x7f22c406e620 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22c37fe640 1 -- 172.21.15.161:0/2034709058 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b8031b30 con 0x7f22c406e620 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22cb102640 1 -- 172.21.15.161:0/2034709058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406e620 msgr2=0x7f22c4070e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22cb102640 1 --2- 172.21.15.161:0/2034709058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406e620 0x7f22c4070e30 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f22b8009870 tx=0x7f22b802b8f0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.432 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22cb102640 1 -- 172.21.15.161:0/2034709058 shutdown_connections 2024-03-23T14:35:06.433 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22cb102640 1 --2- 172.21.15.161:0/2034709058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.433 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22cb102640 1 --2- 172.21.15.161:0/2034709058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406e620 0x7f22c4070e30 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.433 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22cb102640 1 -- 172.21.15.161:0/2034709058 >> 172.21.15.161:0/2034709058 conn(0x7f22c40b1830 msgr2=0x7f22c40b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:06.433 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22cb102640 1 -- 172.21.15.161:0/2034709058 shutdown_connections 2024-03-23T14:35:06.433 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.434+0000 7f22cb102640 1 -- 172.21.15.161:0/2034709058 wait complete. 2024-03-23T14:35:06.436 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.435+0000 7f22cb102640 1 Processor -- start 2024-03-23T14:35:06.436 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.435+0000 7f22cb102640 1 -- start start 2024-03-23T14:35:06.436 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22cb102640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22c406e620 0x7f22c4148640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.436 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22cb102640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406ea00 0x7f22c4144b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.436 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22cb102640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40cbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.436 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22cb102640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22c4158f50 con 0x7f22c406ea00 2024-03-23T14:35:06.436 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22cb102640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f22c414e990 con 0x7f22c406e620 2024-03-23T14:35:06.436 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22cb102640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f22c414ec70 con 0x7f22c4072560 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22c3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406ea00 0x7f22c4144b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22c3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406ea00 0x7f22c4144b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42462/0 (socket says 172.21.15.161:42462) 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.436+0000 7f22c3fff640 1 -- 172.21.15.161:0/1786774912 learned_addr learned my addr 172.21.15.161:0/1786774912 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c8e9e640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22c406e620 0x7f22c4148640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c969f640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40cbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c3fff640 1 -- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 msgr2=0x7f22c40cbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c3fff640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40cbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c3fff640 1 -- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22c406e620 msgr2=0x7f22c4148640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c3fff640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22c406e620 0x7f22c4148640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c3fff640 1 -- 172.21.15.161:0/1786774912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b80094d0 con 0x7f22c406ea00 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c3fff640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406ea00 0x7f22c4144b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f22b000c7f0 tx=0x7f22b000ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c17fa640 1 -- 172.21.15.161:0/1786774912 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0007820 con 0x7f22c406ea00 2024-03-23T14:35:06.437 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c17fa640 1 -- 172.21.15.161:0/1786774912 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b0007980 con 0x7f22c406ea00 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22c414ef50 con 0x7f22c406ea00 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22c17fa640 1 -- 172.21.15.161:0/1786774912 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0007cf0 con 0x7f22c406ea00 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.437+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22c415b370 con 0x7f22c406ea00 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.438+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c41590c0 con 0x7f22c406ea00 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.439+0000 7f22c17fa640 1 -- 172.21.15.161:0/1786774912 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f22b00113f0 con 0x7f22c406ea00 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.439+0000 7f22c17fa640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f22a806b810 0x7f22a806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.439+0000 7f22c17fa640 1 -- 172.21.15.161:0/1786774912 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f22b008a150 con 0x7f22c406ea00 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.439+0000 7f22c8e9e640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f22a806b810 0x7f22a806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.438 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.440+0000 7f22c8e9e640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f22a806b810 0x7f22a806dcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f22b8036000 tx=0x7f22b803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:06.439 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.444+0000 7f22c17fa640 1 -- 172.21.15.161:0/1786774912 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f22b0054f70 con 0x7f22c406ea00 2024-03-23T14:35:06.611 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.615+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22c404f470 con 0x7f22c406ea00 2024-03-23T14:35:06.612 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.616+0000 7f22c17fa640 1 -- 172.21.15.161:0/1786774912 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7f22b0058630 con 0x7f22c406ea00 2024-03-23T14:35:06.612 INFO:teuthology.misc.health.smithi161.stdout: 2024-03-23T14:35:06.612 INFO:teuthology.misc.health.smithi161.stdout:{"epoch":16,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:02.685750+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:06.614 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f22a806b810 msgr2=0x7f22a806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.614 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f22a806b810 0x7f22a806dcd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f22b8036000 tx=0x7f22b803c040 comp rx=0 tx=0).stop 2024-03-23T14:35:06.614 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406ea00 msgr2=0x7f22c4144b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.614 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406ea00 0x7f22c4144b30 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f22b000c7f0 tx=0x7f22b000ccb0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.615 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 shutdown_connections 2024-03-23T14:35:06.615 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f22a806b810 0x7f22a806dcd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.615 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22c4072560 0x7f22c40cbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.615 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22c406ea00 0x7f22c4144b30 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.615 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 --2- 172.21.15.161:0/1786774912 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22c406e620 0x7f22c4148640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.615 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 >> 172.21.15.161:0/1786774912 conn(0x7f22c40b1830 msgr2=0x7f22c40b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:06.615 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 shutdown_connections 2024-03-23T14:35:06.615 INFO:teuthology.misc.health.smithi161.stderr:2024-03-23T14:35:06.618+0000 7f22cb102640 1 -- 172.21.15.161:0/1786774912 wait complete. 2024-03-23T14:35:06.624 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-03-23T14:35:06.624 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-03-23T14:35:06.625 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T14:35:06.625 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T14:35:06.628 INFO:tasks.ceph.mds.a:Started 2024-03-23T14:35:06.628 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T14:35:06.628 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T14:35:06.630 INFO:tasks.ceph.mds.c:Started 2024-03-23T14:35:06.630 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T14:35:06.630 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T14:35:06.633 INFO:tasks.ceph.mds.b:Started 2024-03-23T14:35:06.633 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T14:35:06.633 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T14:35:06.635 INFO:tasks.ceph.mds.d:Started 2024-03-23T14:35:06.635 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-03-23T14:35:06.636 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:06.704 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-23T14:35:06.705 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-23T14:35:06.718 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-23T14:35:06.718 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-23T14:35:06.719 INFO:tasks.ceph.mds.b.smithi183.stderr:2024-03-23T14:35:06.723+0000 7f9a0a91b640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-23T14:35:06.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.746+0000 7efc0ad25640 1 Processor -- start 2024-03-23T14:35:06.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.746+0000 7efc0ad25640 1 -- start start 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc0ad25640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efc040d1470 0x7efc040d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc0ad25640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efc040d2970 0x7efc040b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc0ad25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc040d0460 con 0x7efc0415e400 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc0ad25640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efc040d1050 con 0x7efc040d2970 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc0ad25640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7efc040d2e00 con 0x7efc040d1470 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc092c2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efc040d2970 0x7efc040b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc092c2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efc040d2970 0x7efc040b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42516/0 (socket says 172.21.15.161:42516) 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc092c2640 1 -- 172.21.15.161:0/85070636 learned_addr learned my addr 172.21.15.161:0/85070636 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc03fff640 1 --2- 172.21.15.161:0/85070636 >> v2:172.21.15.183:3300/0 conn(0x7efc040d1470 0x7efc040d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc092c2640 1 -- 172.21.15.161:0/85070636 >> v2:172.21.15.183:3300/0 conn(0x7efc040d1470 msgr2=0x7efc040d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc092c2640 1 --2- 172.21.15.161:0/85070636 >> v2:172.21.15.183:3300/0 conn(0x7efc040d1470 0x7efc040d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc092c2640 1 -- 172.21.15.161:0/85070636 >> v1:172.21.15.183:6789/0 conn(0x7efc0415e400 legacy=0x7efc040cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.747+0000 7efc092c2640 1 -- 172.21.15.161:0/85070636 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc040d06b0 con 0x7efc040d2970 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.748+0000 7efc092c2640 1 --2- 172.21.15.161:0/85070636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d2970 0x7efc040b0a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efc040b6a20 tx=0x7efbfc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fc3f87136c9b09c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.748+0000 7efc037fe640 1 -- 172.21.15.161:0/85070636 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc02cb20 con 0x7efc040d2970 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.748+0000 7efc037fe640 1 -- 172.21.15.161:0/85070636 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbfc02cc80 con 0x7efc040d2970 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.748+0000 7efc037fe640 1 -- 172.21.15.161:0/85070636 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbfc0343f0 con 0x7efc040d2970 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.748+0000 7efc0ad25640 1 -- 172.21.15.161:0/85070636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d2970 msgr2=0x7efc040b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.748+0000 7efc0ad25640 1 --2- 172.21.15.161:0/85070636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d2970 0x7efc040b0a80 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7efc040b6a20 tx=0x7efbfc02bf80 comp rx=0 tx=0).stop 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.749+0000 7efc0ad25640 1 -- 172.21.15.161:0/85070636 shutdown_connections 2024-03-23T14:35:06.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.749+0000 7efc0ad25640 1 --2- 172.21.15.161:0/85070636 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d2970 0x7efc040b0a80 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.749+0000 7efc0ad25640 1 --2- 172.21.15.161:0/85070636 >> v2:172.21.15.183:3300/0 conn(0x7efc040d1470 0x7efc040d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.749+0000 7efc0ad25640 1 -- 172.21.15.161:0/85070636 >> 172.21.15.161:0/85070636 conn(0x7efc0409d190 msgr2=0x7efc040d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:06.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.749+0000 7efc0ad25640 1 -- 172.21.15.161:0/85070636 shutdown_connections 2024-03-23T14:35:06.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.749+0000 7efc0ad25640 1 -- 172.21.15.161:0/85070636 wait complete. 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.750+0000 7efc0ad25640 1 Processor -- start 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.750+0000 7efc0ad25640 1 -- start start 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc0ad25640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d1470 0x7efc041533e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc0ad25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc040d2970 0x7efc040c98d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc0ad25640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc0415e400 0x7efc040c9e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc0ad25640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efc0415fc30 con 0x7efc040d1470 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc0ad25640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efc0415f290 con 0x7efc0415e400 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc0ad25640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efc0415f570 con 0x7efc040d2970 2024-03-23T14:35:06.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d1470 0x7efc041533e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d1470 0x7efc041533e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42522/0 (socket says 172.21.15.161:42522) 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 -- 172.21.15.161:0/1697552452 learned_addr learned my addr 172.21.15.161:0/1697552452 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc092c2640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc0415e400 0x7efc040c9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc03fff640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc040d2970 0x7efc040c98d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 -- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc040d2970 msgr2=0x7efc040c98d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc040d2970 0x7efc040c98d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 -- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc0415e400 msgr2=0x7efc040c9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc0415e400 0x7efc040c9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 -- 172.21.15.161:0/1697552452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efbfc009b30 con 0x7efc040d1470 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc03fff640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc040d2970 0x7efc040c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc08ac1640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d1470 0x7efc041533e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efbf800c400 tx=0x7efbf800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc017fa640 1 -- 172.21.15.161:0/1697552452 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf80052d0 con 0x7efc040d1470 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc017fa640 1 -- 172.21.15.161:0/1697552452 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efbf8005450 con 0x7efc040d1470 2024-03-23T14:35:06.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc0415f850 con 0x7efc040d1470 2024-03-23T14:35:06.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc017fa640 1 -- 172.21.15.161:0/1697552452 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efbf8007260 con 0x7efc040d1470 2024-03-23T14:35:06.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.751+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc041615e0 con 0x7efc040d1470 2024-03-23T14:35:06.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.752+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc040a1e00 con 0x7efc040d1470 2024-03-23T14:35:06.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.753+0000 7efc017fa640 1 -- 172.21.15.161:0/1697552452 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efbf80073c0 con 0x7efc040d1470 2024-03-23T14:35:06.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.753+0000 7efc017fa640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efbe806b810 0x7efbe806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:06.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.753+0000 7efc017fa640 1 -- 172.21.15.161:0/1697552452 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7efbf80891f0 con 0x7efc040d1470 2024-03-23T14:35:06.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.753+0000 7efc03fff640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efbe806b810 0x7efbe806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:06.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.754+0000 7efc03fff640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efbe806b810 0x7efbe806dcd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efbf0006fd0 tx=0x7efbf0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:06.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.757+0000 7efc017fa640 1 -- 172.21.15.161:0/1697552452 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efbf8054010 con 0x7efc040d1470 2024-03-23T14:35:06.879 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:35:06.882 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:35:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.931+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc0415fea0 con 0x7efc040d1470 2024-03-23T14:35:06.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.932+0000 7efc017fa640 1 -- 172.21.15.161:0/1697552452 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7efbf80576d0 con 0x7efc040d1470 2024-03-23T14:35:06.928 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:06.928 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":16,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:02.685750+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:06.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efbe806b810 msgr2=0x7efbe806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efbe806b810 0x7efbe806dcd0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7efbf0006fd0 tx=0x7efbf0008040 comp rx=0 tx=0).stop 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d1470 msgr2=0x7efc041533e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d1470 0x7efc041533e0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7efbf800c400 tx=0x7efbf800c8c0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 shutdown_connections 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efbe806b810 0x7efbe806dcd0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc0415e400 0x7efc040c9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc040d2970 0x7efc040c98d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 --2- 172.21.15.161:0/1697552452 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc040d1470 0x7efc041533e0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 >> 172.21.15.161:0/1697552452 conn(0x7efc0409d190 msgr2=0x7efc040b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 shutdown_connections 2024-03-23T14:35:06.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:06.934+0000 7efc0ad25640 1 -- 172.21.15.161:0/1697552452 wait complete. 2024-03-23T14:35:06.941 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:06.941 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.089+0000 7ff9db88f640 1 Processor -- start 2024-03-23T14:35:07.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.089+0000 7ff9db88f640 1 -- start start 2024-03-23T14:35:07.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9db88f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff9d415e390 0x7ff9d40d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9db88f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 0x7ff9d40d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9db88f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9d40cc4f0 con 0x7ff9d415dfb0 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9db88f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff9d40ccab0 con 0x7ff9d415e390 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9db88f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d40d14c0 con 0x7ff9d40d2520 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9d8e2a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff9d415e390 0x7ff9d40d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9d8e2a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff9d415e390 0x7ff9d40d0a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42528/0 (socket says 172.21.15.161:42528) 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9d8e2a640 1 -- 172.21.15.161:0/3914825514 learned_addr learned my addr 172.21.15.161:0/3914825514 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.090+0000 7ff9d9e2c640 1 --2- 172.21.15.161:0/3914825514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 0x7ff9d40d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.091+0000 7ff9d8e2a640 1 -- 172.21.15.161:0/3914825514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 msgr2=0x7ff9d40d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.091+0000 7ff9d8e2a640 1 --2- 172.21.15.161:0/3914825514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 0x7ff9d40d0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.091+0000 7ff9d8e2a640 1 -- 172.21.15.161:0/3914825514 >> v1:172.21.15.183:6789/0 conn(0x7ff9d415dfb0 legacy=0x7ff9d40d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.091+0000 7ff9d8e2a640 1 -- 172.21.15.161:0/3914825514 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9d40d1770 con 0x7ff9d415e390 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.091+0000 7ff9d8e2a640 1 --2- 172.21.15.161:0/3914825514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40d0a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009870 tx=0x7ff9c402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe1a406f62938583 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.091+0000 7ff9cbfff640 1 -- 172.21.15.161:0/3914825514 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4030840 con 0x7ff9d415e390 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.091+0000 7ff9cbfff640 1 -- 172.21.15.161:0/3914825514 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9c40309a0 con 0x7ff9d415e390 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.091+0000 7ff9cbfff640 1 -- 172.21.15.161:0/3914825514 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9c4030d30 con 0x7ff9d415e390 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.092+0000 7ff9db88f640 1 -- 172.21.15.161:0/3914825514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 msgr2=0x7ff9d40d0a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.092+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3914825514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40d0a40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff9c4009870 tx=0x7ff9c402b980 comp rx=0 tx=0).stop 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.092+0000 7ff9db88f640 1 -- 172.21.15.161:0/3914825514 shutdown_connections 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.092+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3914825514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 0x7ff9d40d0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.092+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3914825514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40d0a40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.092+0000 7ff9db88f640 1 -- 172.21.15.161:0/3914825514 >> 172.21.15.161:0/3914825514 conn(0x7ff9d409aa40 msgr2=0x7ff9d409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:07.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.092+0000 7ff9db88f640 1 -- 172.21.15.161:0/3914825514 shutdown_connections 2024-03-23T14:35:07.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.092+0000 7ff9db88f640 1 -- 172.21.15.161:0/3914825514 wait complete. 2024-03-23T14:35:07.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.093+0000 7ff9db88f640 1 Processor -- start 2024-03-23T14:35:07.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.093+0000 7ff9db88f640 1 -- start start 2024-03-23T14:35:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9db88f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 0x7ff9d41f7020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9db88f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d415dfb0 0x7ff9d40c9090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9db88f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40c7170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9db88f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d40c95d0 con 0x7ff9d415e390 2024-03-23T14:35:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9db88f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff9d41f7f50 con 0x7ff9d415dfb0 2024-03-23T14:35:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9db88f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff9d41f76b0 con 0x7ff9d40d2520 2024-03-23T14:35:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d9e2c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40c7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d9e2c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40c7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42544/0 (socket says 172.21.15.161:42544) 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d9e2c640 1 -- 172.21.15.161:0/3910321358 learned_addr learned my addr 172.21.15.161:0/3910321358 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d962b640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 0x7ff9d41f7020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d8e2a640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d415dfb0 0x7ff9d40c9090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d9e2c640 1 -- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 msgr2=0x7ff9d41f7020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d9e2c640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 0x7ff9d41f7020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d9e2c640 1 -- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d415dfb0 msgr2=0x7ff9d40c9090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d9e2c640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d415dfb0 0x7ff9d40c9090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.094+0000 7ff9d9e2c640 1 -- 172.21.15.161:0/3910321358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9c40094d0 con 0x7ff9d415e390 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.095+0000 7ff9d9e2c640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40c7170 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc004650 tx=0x7ff9cc00ece0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.095+0000 7ff9c9ffb640 1 -- 172.21.15.161:0/3910321358 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc009630 con 0x7ff9d415e390 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.095+0000 7ff9c9ffb640 1 -- 172.21.15.161:0/3910321358 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff9cc00c940 con 0x7ff9d415e390 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.095+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9d41f7990 con 0x7ff9d415e390 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.095+0000 7ff9c9ffb640 1 -- 172.21.15.161:0/3910321358 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff9cc00ccf0 con 0x7ff9d415e390 2024-03-23T14:35:07.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.095+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9d41f3bd0 con 0x7ff9d415e390 2024-03-23T14:35:07.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.096+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9d41f3a30 con 0x7ff9d415e390 2024-03-23T14:35:07.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.096+0000 7ff9c9ffb640 1 -- 172.21.15.161:0/3910321358 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff9cc013470 con 0x7ff9d415e390 2024-03-23T14:35:07.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.097+0000 7ff9c9ffb640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff9b806b810 0x7ff9b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.097+0000 7ff9c9ffb640 1 -- 172.21.15.161:0/3910321358 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ff9cc08a000 con 0x7ff9d415e390 2024-03-23T14:35:07.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.097+0000 7ff9d962b640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff9b806b810 0x7ff9b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.097+0000 7ff9d962b640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff9b806b810 0x7ff9b806dcd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40c7c40 tx=0x7ff9d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.101+0000 7ff9c9ffb640 1 -- 172.21.15.161:0/3910321358 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff9d41f3a30 con 0x7ff9d415e390 2024-03-23T14:35:07.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.274+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff9d41f3a30 con 0x7ff9d415e390 2024-03-23T14:35:07.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.274+0000 7ff9c9ffb640 1 -- 172.21.15.161:0/3910321358 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7ff9d41f3a30 con 0x7ff9d415e390 2024-03-23T14:35:07.271 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff9b806b810 msgr2=0x7ff9b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff9b806b810 0x7ff9b806dcd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff9d40c7c40 tx=0x7ff9d0009040 comp rx=0 tx=0).stop 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 msgr2=0x7ff9d40c7170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40c7170 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff9cc004650 tx=0x7ff9cc00ece0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 shutdown_connections 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff9b806b810 0x7ff9b806dcd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff9d415e390 0x7ff9d40c7170 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff9d415dfb0 0x7ff9d40c9090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 --2- 172.21.15.161:0/3910321358 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff9d40d2520 0x7ff9d41f7020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.276+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 >> 172.21.15.161:0/3910321358 conn(0x7ff9d409aa40 msgr2=0x7ff9d406d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.277+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 shutdown_connections 2024-03-23T14:35:07.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.277+0000 7ff9db88f640 1 -- 172.21.15.161:0/3910321358 wait complete. 2024-03-23T14:35:07.283 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:07.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.388+0000 7feb71c0b640 1 Processor -- start 2024-03-23T14:35:07.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.388+0000 7feb71c0b640 1 -- start start 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb71c0b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c06eca0 0x7feb6c165d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb71c0b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c15be70 con 0x7feb6c072100 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb71c0b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb6c15c150 con 0x7feb6c0724e0 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb71c0b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feb6c15c430 con 0x7feb6c06eca0 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7feb6c072100 0x7feb6c15b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:33810/0 (socket says 172.21.15.161:33810) 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6b7fe640 1 -- 172.21.15.161:0/1277572716 learned_addr learned my addr 172.21.15.161:0/1277572716 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6bfff640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c06eca0 0x7feb6c165d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 944771130 0 0) 0x7feb6c15be70 con 0x7feb6c072100 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb54003520 con 0x7feb6c072100 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104210976 0 0) 0x7feb6c15c150 con 0x7feb6c0724e0 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb6c15be70 con 0x7feb6c0724e0 2024-03-23T14:35:07.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 606122030 0 0) 0x7feb54003520 con 0x7feb6c072100 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c06eca0 msgr2=0x7feb6c165d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c06eca0 0x7feb6c165d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 >> v1:172.21.15.183:6789/0 conn(0x7feb6c0724e0 legacy=0x7feb6c06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c0b2c70 con 0x7feb6c072100 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7feb5c001670 con 0x7feb6c072100 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 >> v1:172.21.15.161:6789/0 conn(0x7feb6c072100 legacy=0x7feb6c15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c06eca0 0x7feb54003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb540047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb54004d00 0x7feb54005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c15c430 con 0x7feb54004090 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feb5400ba60 con 0x7feb54004d00 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feb5400bc20 con 0x7feb6c06eca0 2024-03-23T14:35:07.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6affd640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb540047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6b7fe640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c06eca0 0x7feb54003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.389+0000 7feb6bfff640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb54004d00 0x7feb54005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6affd640 1 -- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c06eca0 msgr2=0x7feb54003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6affd640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c06eca0 0x7feb54003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6affd640 1 -- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb54004d00 msgr2=0x7feb54005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6affd640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb54004d00 0x7feb54005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6affd640 1 -- 172.21.15.161:0/1277572716 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb54003520 con 0x7feb54004090 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6affd640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb540047c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feb600018a0 tx=0x7feb540057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=648f075d421e6746 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb600052d0 con 0x7feb54004090 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb60005430 con 0x7feb54004090 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.390+0000 7feb6a7fc640 1 -- 172.21.15.161:0/1277572716 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb60003810 con 0x7feb54004090 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.391+0000 7feb71c0b640 1 -- 172.21.15.161:0/1277572716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 msgr2=0x7feb540047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.391+0000 7feb71c0b640 1 --2- 172.21.15.161:0/1277572716 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb540047c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7feb600018a0 tx=0x7feb540057f0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.391+0000 7feb6bfff640 1 -- 172.21.15.161:0/1277572716 reap_dead start 2024-03-23T14:35:07.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.391+0000 7feb71c0b640 1 -- 172.21.15.161:0/1277572716 shutdown_connections 2024-03-23T14:35:07.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.391+0000 7feb71c0b640 1 -- 172.21.15.161:0/1277572716 >> 172.21.15.161:0/1277572716 conn(0x7feb6c0b17f0 msgr2=0x7feb6c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:07.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.391+0000 7feb71c0b640 1 -- 172.21.15.161:0/1277572716 shutdown_connections 2024-03-23T14:35:07.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.391+0000 7feb71c0b640 1 -- 172.21.15.161:0/1277572716 wait complete. 2024-03-23T14:35:07.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.392+0000 7feb71c0b640 1 Processor -- start 2024-03-23T14:35:07.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.392+0000 7feb71c0b640 1 -- start start 2024-03-23T14:35:07.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb71c0b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb6c158ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb71c0b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c0a9750 0x7feb6c145eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb71c0b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c15a460 0x7feb6c15a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb71c0b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c159fd0 con 0x7feb54004090 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb71c0b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feb6c15ae30 con 0x7feb6c15a460 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb71c0b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feb6c1593c0 con 0x7feb6c0a9750 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb6c158ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb6c158ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42566/0 (socket says 172.21.15.161:42566) 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6b7fe640 1 -- 172.21.15.161:0/2554545709 learned_addr learned my addr 172.21.15.161:0/2554545709 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6affd640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c0a9750 0x7feb6c145eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6bfff640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c15a460 0x7feb6c15a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6b7fe640 1 -- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c0a9750 msgr2=0x7feb6c145eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6b7fe640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c0a9750 0x7feb6c145eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6b7fe640 1 -- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c15a460 msgr2=0x7feb6c15a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6b7fe640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c15a460 0x7feb6c15a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.393+0000 7feb6b7fe640 1 -- 172.21.15.161:0/2554545709 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c1596a0 con 0x7feb54004090 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.394+0000 7feb6b7fe640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb6c158ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb5c007310 tx=0x7feb5c0077d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:07.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.394+0000 7feb53fff640 1 -- 172.21.15.161:0/2554545709 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c00f960 con 0x7feb54004090 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.394+0000 7feb53fff640 1 -- 172.21.15.161:0/2554545709 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb5c00fac0 con 0x7feb54004090 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.394+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb6c159980 con 0x7feb54004090 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.394+0000 7feb53fff640 1 -- 172.21.15.161:0/2554545709 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb5c00fe50 con 0x7feb54004090 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.394+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb6c157750 con 0x7feb54004090 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.395+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb6c0735e0 con 0x7feb54004090 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.395+0000 7feb53fff640 1 -- 172.21.15.161:0/2554545709 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feb5c01d460 con 0x7feb54004090 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.396+0000 7feb53fff640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb5806b810 0x7feb5806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.396+0000 7feb53fff640 1 -- 172.21.15.161:0/2554545709 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7feb5c089f30 con 0x7feb54004090 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.396+0000 7feb6affd640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb5806b810 0x7feb5806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.396+0000 7feb6affd640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb5806b810 0x7feb5806dcd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb60034000 tx=0x7feb60031000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:07.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.400+0000 7feb53fff640 1 -- 172.21.15.161:0/2554545709 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feb5c054d50 con 0x7feb54004090 2024-03-23T14:35:07.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.610+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feb6c04f550 con 0x7feb54004090 2024-03-23T14:35:07.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.611+0000 7feb53fff640 1 -- 172.21.15.161:0/2554545709 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7feb5c058410 con 0x7feb54004090 2024-03-23T14:35:07.607 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:07.607 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-03-23T14:35:07.607 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 2 2024-03-23T14:35:07.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.612+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb5806b810 msgr2=0x7feb5806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.612+0000 7feb71c0b640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb5806b810 0x7feb5806dcd0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb60034000 tx=0x7feb60031000 comp rx=0 tx=0).stop 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.612+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 msgr2=0x7feb6c158ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.612+0000 7feb71c0b640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb6c158ac0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7feb5c007310 tx=0x7feb5c0077d0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.613+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 shutdown_connections 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.613+0000 7feb71c0b640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb5806b810 0x7feb5806dcd0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.613+0000 7feb71c0b640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c15a460 0x7feb6c15a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.613+0000 7feb71c0b640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb6c0a9750 0x7feb6c145eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.613+0000 7feb71c0b640 1 --2- 172.21.15.161:0/2554545709 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb54004090 0x7feb6c158ac0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.613+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 >> 172.21.15.161:0/2554545709 conn(0x7feb6c0b17f0 msgr2=0x7feb6c0cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.613+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 shutdown_connections 2024-03-23T14:35:07.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:07.613+0000 7feb71c0b640 1 -- 172.21.15.161:0/2554545709 wait complete. 2024-03-23T14:35:09.620 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:09.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.736+0000 7fcece6e8640 1 Processor -- start 2024-03-23T14:35:09.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.736+0000 7fcece6e8640 1 -- start start 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fcece6e8640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcec815e650 0x7fcec80d04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fcece6e8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcec80cb230 0x7fcec80cb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fcece6e8640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec80d0a20 con 0x7fcec815be00 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fcece6e8640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcec815f5f0 con 0x7fcec80cb230 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fcece6e8640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fcec815ea90 con 0x7fcec815e650 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fceccc85640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcec80cb230 0x7fcec80cb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fcec7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fcec815be00 0x7fcec8162300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:33826/0 (socket says 172.21.15.161:33826) 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fceccc85640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcec80cb230 0x7fcec80cb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42572/0 (socket says 172.21.15.161:42572) 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fcec7fff640 1 -- 172.21.15.161:0/529191981 learned_addr learned my addr 172.21.15.161:0/529191981 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.737+0000 7fcec77fe640 1 --2- 172.21.15.161:0/529191981 >> v2:172.21.15.183:3300/0 conn(0x7fcec815e650 0x7fcec80d04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.738+0000 7fceccc85640 1 -- 172.21.15.161:0/529191981 >> v2:172.21.15.183:3300/0 conn(0x7fcec815e650 msgr2=0x7fcec80d04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:09.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.738+0000 7fceccc85640 1 --2- 172.21.15.161:0/529191981 >> v2:172.21.15.183:3300/0 conn(0x7fcec815e650 0x7fcec80d04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.738+0000 7fceccc85640 1 -- 172.21.15.161:0/529191981 >> v1:172.21.15.161:6789/0 conn(0x7fcec815be00 legacy=0x7fcec8162300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:09.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.738+0000 7fcec6ffd640 1 -- 172.21.15.161:0/529191981 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679048347 0 0) 0x7fcec80d0a20 con 0x7fcec815be00 2024-03-23T14:35:09.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.738+0000 7fceccc85640 1 -- 172.21.15.161:0/529191981 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec815ed50 con 0x7fcec80cb230 2024-03-23T14:35:09.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.738+0000 7fceccc85640 1 --2- 172.21.15.161:0/529191981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec80cb230 0x7fcec80cb6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcec80738f0 tx=0x7fcec002bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80e049a00686c6e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:09.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcec6ffd640 1 -- 172.21.15.161:0/529191981 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec002cc80 con 0x7fcec80cb230 2024-03-23T14:35:09.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcec6ffd640 1 -- 172.21.15.161:0/529191981 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcec002cde0 con 0x7fcec80cb230 2024-03-23T14:35:09.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcec6ffd640 1 -- 172.21.15.161:0/529191981 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec0034660 con 0x7fcec80cb230 2024-03-23T14:35:09.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcece6e8640 1 -- 172.21.15.161:0/529191981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec80cb230 msgr2=0x7fcec80cb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:09.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcece6e8640 1 --2- 172.21.15.161:0/529191981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec80cb230 0x7fcec80cb6c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fcec80738f0 tx=0x7fcec002bed0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcece6e8640 1 -- 172.21.15.161:0/529191981 shutdown_connections 2024-03-23T14:35:09.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcece6e8640 1 --2- 172.21.15.161:0/529191981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec80cb230 0x7fcec80cb6c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcece6e8640 1 --2- 172.21.15.161:0/529191981 >> v2:172.21.15.183:3300/0 conn(0x7fcec815e650 0x7fcec80d04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcece6e8640 1 -- 172.21.15.161:0/529191981 >> 172.21.15.161:0/529191981 conn(0x7fcec80d0be0 msgr2=0x7fcec80d0ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:09.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcece6e8640 1 -- 172.21.15.161:0/529191981 shutdown_connections 2024-03-23T14:35:09.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.739+0000 7fcece6e8640 1 -- 172.21.15.161:0/529191981 wait complete. 2024-03-23T14:35:09.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.740+0000 7fcece6e8640 1 Processor -- start 2024-03-23T14:35:09.737 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fcece6e8640 1 -- start start 2024-03-23T14:35:09.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fcece6e8640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcec80cb230 0x7fcec80c9480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:09.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fcece6e8640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcec815be00 0x7fcec815b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fcece6e8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec815e650 0x7fcec80aefe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fcece6e8640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcec806ed70 con 0x7fcec815e650 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fcece6e8640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcec80c99c0 con 0x7fcec80cb230 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fcece6e8640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fcec80b2220 con 0x7fcec815be00 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fceccc85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec815e650 0x7fcec80aefe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fceccc85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec815e650 0x7fcec80aefe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42582/0 (socket says 172.21.15.161:42582) 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fceccc85640 1 -- 172.21.15.161:0/3330591619 learned_addr learned my addr 172.21.15.161:0/3330591619 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.741+0000 7fcec7fff640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcec80cb230 0x7fcec80c9480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fcec77fe640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcec815be00 0x7fcec815b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fceccc85640 1 -- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcec815be00 msgr2=0x7fcec815b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fceccc85640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcec815be00 0x7fcec815b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fceccc85640 1 -- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcec80cb230 msgr2=0x7fcec80c9480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fceccc85640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcec80cb230 0x7fcec80c9480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fceccc85640 1 -- 172.21.15.161:0/3330591619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec0009a70 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fceccc85640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec815e650 0x7fcec80aefe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcec002c4e0 tx=0x7fcec0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fcec4ff9640 1 -- 172.21.15.161:0/3330591619 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec003e610 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fcec4ff9640 1 -- 172.21.15.161:0/3330591619 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcec003e770 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fcec4ff9640 1 -- 172.21.15.161:0/3330591619 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcec003eb20 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec815b6d0 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.742+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec80c86c0 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.743+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec804f550 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.743+0000 7fcec4ff9640 1 -- 172.21.15.161:0/3330591619 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcec003b050 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.744+0000 7fcec4ff9640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fceb406b810 0x7fceb406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.744+0000 7fcec4ff9640 1 -- 172.21.15.161:0/3330591619 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7fcec00ac620 con 0x7fcec815e650 2024-03-23T14:35:09.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.744+0000 7fcec7fff640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fceb406b810 0x7fceb406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:09.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.744+0000 7fcec7fff640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fceb406b810 0x7fceb406dcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fceb8000f60 tx=0x7fceb8007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:09.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.748+0000 7fcec4ff9640 1 -- 172.21.15.161:0/3330591619 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcec0077440 con 0x7fcec815e650 2024-03-23T14:35:09.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.956+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcec806bdd0 con 0x7fcec815e650 2024-03-23T14:35:09.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.957+0000 7fcec4ff9640 1 -- 172.21.15.161:0/3330591619 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+3870 (secure 0 0 0) 0x7fcec0039090 con 0x7fcec815e650 2024-03-23T14:35:09.953 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:09.953 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4282,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-03-23T14:35:09.953 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 3 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fceb406b810 msgr2=0x7fceb406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fceb406b810 0x7fceb406dcd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fceb8000f60 tx=0x7fceb8007d50 comp rx=0 tx=0).stop 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec815e650 msgr2=0x7fcec80aefe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec815e650 0x7fcec80aefe0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fcec002c4e0 tx=0x7fcec0004a40 comp rx=0 tx=0).stop 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 shutdown_connections 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fceb406b810 0x7fceb406dcd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcec815e650 0x7fcec80aefe0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcec815be00 0x7fcec815b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 --2- 172.21.15.161:0/3330591619 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcec80cb230 0x7fcec80c9480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 >> 172.21.15.161:0/3330591619 conn(0x7fcec80d0be0 msgr2=0x7fcec806b080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 shutdown_connections 2024-03-23T14:35:09.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:09.959+0000 7fcece6e8640 1 -- 172.21.15.161:0/3330591619 wait complete. 2024-03-23T14:35:09.965 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:10.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.069+0000 7f0c441a1640 1 Processor -- start 2024-03-23T14:35:10.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c441a1640 1 -- start start 2024-03-23T14:35:10.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c441a1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0c3c15e400 0x7f0c3c0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c441a1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0c3c0d2b40 0x7f0c3c0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c441a1640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c3c0d0460 con 0x7f0c3c0cf9d0 2024-03-23T14:35:10.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c441a1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0c3c0d1050 con 0x7f0c3c15e400 2024-03-23T14:35:10.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c441a1640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f0c3c0d06b0 con 0x7f0c3c0d2b40 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c41f3d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0c3c15e400 0x7f0c3c0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c4173c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0c3c0cf9d0 0x7f0c3c0d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:33842/0 (socket says 172.21.15.161:33842) 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.070+0000 7f0c4173c640 1 -- 172.21.15.161:0/2707372053 learned_addr learned my addr 172.21.15.161:0/2707372053 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.071+0000 7f0c4273e640 1 --2- 172.21.15.161:0/2707372053 >> v2:172.21.15.183:3300/0 conn(0x7f0c3c0d2b40 0x7f0c3c0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.071+0000 7f0c41f3d640 1 -- 172.21.15.161:0/2707372053 >> v2:172.21.15.183:3300/0 conn(0x7f0c3c0d2b40 msgr2=0x7f0c3c0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.071+0000 7f0c41f3d640 1 --2- 172.21.15.161:0/2707372053 >> v2:172.21.15.183:3300/0 conn(0x7f0c3c0d2b40 0x7f0c3c0b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.071+0000 7f0c41f3d640 1 -- 172.21.15.161:0/2707372053 >> v1:172.21.15.161:6789/0 conn(0x7f0c3c0cf9d0 legacy=0x7f0c3c0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.071+0000 7f0c41f3d640 1 -- 172.21.15.161:0/2707372053 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c3c0d0990 con 0x7f0c3c15e400 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.071+0000 7f0c40f3b640 1 -- 172.21.15.161:0/2707372053 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2745748835 0 0) 0x7f0c3c0d0460 con 0x7f0c3c0cf9d0 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.071+0000 7f0c4273e640 1 --2- 172.21.15.161:0/2707372053 >> v2:172.21.15.183:3300/0 conn(0x7f0c3c0d2b40 0x7f0c3c0b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.071+0000 7f0c41f3d640 1 --2- 172.21.15.161:0/2707372053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0cf490 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0ce430 tx=0x7f0c3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3410d5f5aa5b322b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c40f3b640 1 -- 172.21.15.161:0/2707372053 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c3802cc80 con 0x7f0c3c15e400 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c40f3b640 1 -- 172.21.15.161:0/2707372053 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c3802cde0 con 0x7f0c3c15e400 2024-03-23T14:35:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c40f3b640 1 -- 172.21.15.161:0/2707372053 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c38034660 con 0x7f0c3c15e400 2024-03-23T14:35:10.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c441a1640 1 -- 172.21.15.161:0/2707372053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 msgr2=0x7f0c3c0cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2707372053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0cf490 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c0ce430 tx=0x7f0c3802bed0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c441a1640 1 -- 172.21.15.161:0/2707372053 shutdown_connections 2024-03-23T14:35:10.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2707372053 >> v2:172.21.15.183:3300/0 conn(0x7f0c3c0d2b40 0x7f0c3c0b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2707372053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0cf490 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c441a1640 1 -- 172.21.15.161:0/2707372053 >> 172.21.15.161:0/2707372053 conn(0x7f0c3c09d190 msgr2=0x7f0c3c0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:10.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c441a1640 1 -- 172.21.15.161:0/2707372053 shutdown_connections 2024-03-23T14:35:10.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.072+0000 7f0c441a1640 1 -- 172.21.15.161:0/2707372053 wait complete. 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.073+0000 7f0c441a1640 1 Processor -- start 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.073+0000 7f0c441a1640 1 -- start start 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c441a1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c3c0cf9d0 0x7f0c3c14a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c441a1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c3c0d2b40 0x7f0c3c0c9420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c441a1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0c9960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c441a1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c0c89d0 con 0x7f0c3c15e400 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c441a1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0c3c0cd9d0 con 0x7f0c3c0cf9d0 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c441a1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0c3c0cdcb0 con 0x7f0c3c0d2b40 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c4273e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0c9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c4273e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0c9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42602/0 (socket says 172.21.15.161:42602) 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c4273e640 1 -- 172.21.15.161:0/2452592990 learned_addr learned my addr 172.21.15.161:0/2452592990 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c41f3d640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c3c0cf9d0 0x7f0c3c14a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c4273e640 1 -- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c3c0d2b40 msgr2=0x7f0c3c0c9420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.074+0000 7f0c4273e640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c3c0d2b40 0x7f0c3c0c9420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c4273e640 1 -- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c3c0cf9d0 msgr2=0x7f0c3c14a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c4273e640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c3c0cf9d0 0x7f0c3c14a970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c4273e640 1 -- 172.21.15.161:0/2452592990 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c38009a70 con 0x7f0c3c15e400 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c4273e640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0c9960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400e5e0 tx=0x7f0c3400eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c327fc640 1 -- 172.21.15.161:0/2452592990 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34004230 con 0x7f0c3c15e400 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c327fc640 1 -- 172.21.15.161:0/2452592990 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c34004390 con 0x7f0c3c15e400 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c3c0c8c20 con 0x7f0c3c15e400 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c327fc640 1 -- 172.21.15.161:0/2452592990 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c34003830 con 0x7f0c3c15e400 2024-03-23T14:35:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.075+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c3c14aeb0 con 0x7f0c3c15e400 2024-03-23T14:35:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.076+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c3c0d1450 con 0x7f0c3c15e400 2024-03-23T14:35:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.076+0000 7f0c327fc640 1 -- 172.21.15.161:0/2452592990 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0c34003990 con 0x7f0c3c15e400 2024-03-23T14:35:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.077+0000 7f0c327fc640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0c2006b810 0x7f0c2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.077+0000 7f0c327fc640 1 -- 172.21.15.161:0/2452592990 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f0c34089a40 con 0x7f0c3c15e400 2024-03-23T14:35:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.077+0000 7f0c41f3d640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0c2006b810 0x7f0c2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.077+0000 7f0c41f3d640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0c2006b810 0x7f0c2006dcd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c38036000 tx=0x7f0c38030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.081+0000 7f0c327fc640 1 -- 172.21.15.161:0/2452592990 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0c3c0d1450 con 0x7f0c3c15e400 2024-03-23T14:35:10.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.253+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0c3c0c9140 con 0x7f0c3c15e400 2024-03-23T14:35:10.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.254+0000 7f0c327fc640 1 -- 172.21.15.161:0/2452592990 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) ==== 74+0+12920 (secure 0 0 0) 0x7f0c3c0c9140 con 0x7f0c3c15e400 2024-03-23T14:35:10.250 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:10.250 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":16,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:02.685750+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:10.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0c2006b810 msgr2=0x7f0c2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0c2006b810 0x7f0c2006dcd0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f0c38036000 tx=0x7f0c38030000 comp rx=0 tx=0).stop 2024-03-23T14:35:10.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 msgr2=0x7f0c3c0c9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0c9960 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0c3400e5e0 tx=0x7f0c3400eaa0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 shutdown_connections 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0c2006b810 0x7f0c2006dcd0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c3c15e400 0x7f0c3c0c9960 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c3c0d2b40 0x7f0c3c0c9420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 --2- 172.21.15.161:0/2452592990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c3c0cf9d0 0x7f0c3c14a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 >> 172.21.15.161:0/2452592990 conn(0x7f0c3c09d190 msgr2=0x7f0c3c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 shutdown_connections 2024-03-23T14:35:10.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.256+0000 7f0c441a1640 1 -- 172.21.15.161:0/2452592990 wait complete. 2024-03-23T14:35:10.263 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:10.263 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:10.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.370+0000 7f4e97015640 1 Processor -- start 2024-03-23T14:35:10.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.370+0000 7f4e97015640 1 -- start start 2024-03-23T14:35:10.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e97015640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4e9015e390 0x7f4e900d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e97015640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e900d2520 0x7f4e900d0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e97015640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e900cc4f0 con 0x7f4e9015dfb0 2024-03-23T14:35:10.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e97015640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f4e900ccab0 con 0x7f4e9015e390 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e97015640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4e900d14c0 con 0x7f4e900d2520 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e94db1640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f4e9015dfb0 0x7f4e900d1e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:33852/0 (socket says 172.21.15.161:33852) 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e94db1640 1 -- 172.21.15.161:0/1473017925 learned_addr learned my addr 172.21.15.161:0/1473017925 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3535543910 0 0) 0x7f4e900cc4f0 con 0x7f4e9015dfb0 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4e78003520 con 0x7f4e9015dfb0 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e87fff640 1 --2- 172.21.15.161:0/1473017925 >> v2:172.21.15.183:3300/0 conn(0x7f4e9015e390 0x7f4e900d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e955b2640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e900d2520 0x7f4e900d0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1350459112 0 0) 0x7f4e78003520 con 0x7f4e9015dfb0 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e900d2520 msgr2=0x7f4e900d0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e900d2520 0x7f4e900d0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 >> v2:172.21.15.183:3300/0 conn(0x7f4e9015e390 msgr2=0x7f4e900d0a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 --2- 172.21.15.161:0/1473017925 >> v2:172.21.15.183:3300/0 conn(0x7f4e9015e390 0x7f4e900d0a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e900d1770 con 0x7f4e9015dfb0 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f4e8c001670 con 0x7f4e9015dfb0 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.371+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 >> v1:172.21.15.161:6789/0 conn(0x7f4e9015dfb0 legacy=0x7f4e900d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e877fe640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e78003970 0x7f4e78003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e877fe640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e780047c0 0x7f4e78004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e877fe640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e900d2520 0x7f4e78005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4e900ccab0 con 0x7f4e78003970 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4e900d14c0 con 0x7f4e780047c0 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4e7800ba00 con 0x7f4e900d2520 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e94db1640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e78003970 0x7f4e78003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e87fff640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e780047c0 0x7f4e78004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e955b2640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e900d2520 0x7f4e78005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e94db1640 1 -- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e900d2520 msgr2=0x7f4e78005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e94db1640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e900d2520 0x7f4e78005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e94db1640 1 -- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e780047c0 msgr2=0x7f4e78004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e94db1640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e780047c0 0x7f4e78004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.372+0000 7f4e94db1640 1 -- 172.21.15.161:0/1473017925 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e78003520 con 0x7f4e78003970 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e94db1640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e78003970 0x7f4e78003d90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00c210 tx=0x7f4e780057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71cd50b075a5617f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c005ad0 con 0x7f4e78003970 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e8c005c30 con 0x7f4e78003970 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e877fe640 1 -- 172.21.15.161:0/1473017925 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e8c02c810 con 0x7f4e78003970 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e97015640 1 -- 172.21.15.161:0/1473017925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e78003970 msgr2=0x7f4e78003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e97015640 1 --2- 172.21.15.161:0/1473017925 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e78003970 0x7f4e78003d90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c00c210 tx=0x7f4e780057f0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e955b2640 1 -- 172.21.15.161:0/1473017925 reap_dead start 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e97015640 1 -- 172.21.15.161:0/1473017925 shutdown_connections 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e97015640 1 -- 172.21.15.161:0/1473017925 >> 172.21.15.161:0/1473017925 conn(0x7f4e9009aa40 msgr2=0x7f4e9009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e97015640 1 -- 172.21.15.161:0/1473017925 shutdown_connections 2024-03-23T14:35:10.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.373+0000 7f4e97015640 1 -- 172.21.15.161:0/1473017925 wait complete. 2024-03-23T14:35:10.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.374+0000 7f4e97015640 1 Processor -- start 2024-03-23T14:35:10.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.374+0000 7f4e97015640 1 -- start start 2024-03-23T14:35:10.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e97015640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e78003970 0x7f4e9014e300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e97015640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e900c7d70 0x7f4e900c81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e97015640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e900c8fe0 0x7f4e900c9440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e97015640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4e900c8730 con 0x7f4e900c8fe0 2024-03-23T14:35:10.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e97015640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4e900c5b90 con 0x7f4e900c7d70 2024-03-23T14:35:10.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e97015640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4e900ca5f0 con 0x7f4e78003970 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e955b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e900c8fe0 0x7f4e900c9440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e955b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e900c8fe0 0x7f4e900c9440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42614/0 (socket says 172.21.15.161:42614) 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.375+0000 7f4e955b2640 1 -- 172.21.15.161:0/3332611849 learned_addr learned my addr 172.21.15.161:0/3332611849 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e94db1640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e78003970 0x7f4e9014e300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e87fff640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e900c7d70 0x7f4e900c81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e955b2640 1 -- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e78003970 msgr2=0x7f4e9014e300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e955b2640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e78003970 0x7f4e9014e300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e955b2640 1 -- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e900c7d70 msgr2=0x7f4e900c81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e955b2640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e900c7d70 0x7f4e900c81f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e955b2640 1 -- 172.21.15.161:0/3332611849 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e9014e840 con 0x7f4e900c8fe0 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e955b2640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e900c8fe0 0x7f4e900c9440 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4e900cc4a0 tx=0x7f4e88003e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e857fa640 1 -- 172.21.15.161:0/3332611849 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88007b10 con 0x7f4e900c8fe0 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e857fa640 1 -- 172.21.15.161:0/3332611849 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e88007c70 con 0x7f4e900c8fe0 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e900c9d50 con 0x7f4e900c8fe0 2024-03-23T14:35:10.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e857fa640 1 -- 172.21.15.161:0/3332611849 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e88015420 con 0x7f4e900c8fe0 2024-03-23T14:35:10.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.376+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e900ca310 con 0x7f4e900c8fe0 2024-03-23T14:35:10.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.377+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e9004f4e0 con 0x7f4e900c8fe0 2024-03-23T14:35:10.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.378+0000 7f4e857fa640 1 -- 172.21.15.161:0/3332611849 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4e88015580 con 0x7f4e900c8fe0 2024-03-23T14:35:10.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.378+0000 7f4e857fa640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4e7c06b810 0x7f4e7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.378+0000 7f4e857fa640 1 -- 172.21.15.161:0/3332611849 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f4e8808a530 con 0x7f4e900c8fe0 2024-03-23T14:35:10.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.378+0000 7f4e94db1640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4e7c06b810 0x7f4e7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.378+0000 7f4e94db1640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4e7c06b810 0x7f4e7c06dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c034000 tx=0x7f4e8c007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.382+0000 7f4e857fa640 1 -- 172.21.15.161:0/3332611849 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4e88055330 con 0x7f4e900c8fe0 2024-03-23T14:35:10.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.556+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4e900c5cd0 con 0x7f4e900c8fe0 2024-03-23T14:35:10.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.556+0000 7f4e857fa640 1 -- 172.21.15.161:0/3332611849 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v16) ==== 93+0+10 (secure 0 0 0) 0x7f4e880589f0 con 0x7f4e900c8fe0 2024-03-23T14:35:10.552 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:10.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4e7c06b810 msgr2=0x7f4e7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4e7c06b810 0x7f4e7c06dcd0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4e8c034000 tx=0x7f4e8c007000 comp rx=0 tx=0).stop 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e900c8fe0 msgr2=0x7f4e900c9440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e900c8fe0 0x7f4e900c9440 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f4e900cc4a0 tx=0x7f4e88003e90 comp rx=0 tx=0).stop 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 shutdown_connections 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4e7c06b810 0x7f4e7c06dcd0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4e900c8fe0 0x7f4e900c9440 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4e900c7d70 0x7f4e900c81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 --2- 172.21.15.161:0/3332611849 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4e78003970 0x7f4e9014e300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 >> 172.21.15.161:0/3332611849 conn(0x7f4e9009aa40 msgr2=0x7f4e9015cc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 shutdown_connections 2024-03-23T14:35:10.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.559+0000 7f4e97015640 1 -- 172.21.15.161:0/3332611849 wait complete. 2024-03-23T14:35:10.565 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T14:35:10.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.670+0000 7f0a5ef6c640 1 Processor -- start 2024-03-23T14:35:10.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.670+0000 7f0a5ef6c640 1 -- start start 2024-03-23T14:35:10.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.671+0000 7f0a5ef6c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a580d1270 0x7f0a580d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.671+0000 7f0a5ef6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d29d0 0x7f0a580b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.671+0000 7f0a5ef6c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a58161c00 con 0x7f0a58161820 2024-03-23T14:35:10.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.671+0000 7f0a5ef6c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0a580d2e20 con 0x7f0a580d1270 2024-03-23T14:35:10.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.671+0000 7f0a5ef6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0a580cea30 con 0x7f0a580d29d0 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.671+0000 7f0a57fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a580d1270 0x7f0a580d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.671+0000 7f0a5cd08640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0a58161820 0x7f0a580d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49734/0 (socket says 172.21.15.161:49734) 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.671+0000 7f0a5cd08640 1 -- 172.21.15.161:0/2622437302 learned_addr learned my addr 172.21.15.161:0/2622437302 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.672+0000 7f0a5d509640 1 --2- 172.21.15.161:0/2622437302 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d29d0 0x7f0a580b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.672+0000 7f0a57fff640 1 -- 172.21.15.161:0/2622437302 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d29d0 msgr2=0x7f0a580b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.672+0000 7f0a57fff640 1 --2- 172.21.15.161:0/2622437302 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d29d0 0x7f0a580b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.672+0000 7f0a57fff640 1 -- 172.21.15.161:0/2622437302 >> v1:172.21.15.161:6789/0 conn(0x7f0a58161820 legacy=0x7f0a580d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.672+0000 7f0a577fe640 1 -- 172.21.15.161:0/2622437302 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2098425622 0 0) 0x7f0a58161c00 con 0x7f0a58161820 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.672+0000 7f0a57fff640 1 -- 172.21.15.161:0/2622437302 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a580cfd70 con 0x7f0a580d1270 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.672+0000 7f0a57fff640 1 --2- 172.21.15.161:0/2622437302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a580d1270 0x7f0a580d2490 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a400097a0 tx=0x7f0a4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfbfe1325db76619 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a577fe640 1 -- 172.21.15.161:0/2622437302 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40031830 con 0x7f0a580d1270 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a577fe640 1 -- 172.21.15.161:0/2622437302 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a40031990 con 0x7f0a580d1270 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a577fe640 1 -- 172.21.15.161:0/2622437302 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a40031d20 con 0x7f0a580d1270 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/2622437302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a580d1270 msgr2=0x7f0a580d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/2622437302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a580d1270 0x7f0a580d2490 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f0a400097a0 tx=0x7f0a4002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/2622437302 shutdown_connections 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/2622437302 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d29d0 0x7f0a580b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/2622437302 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a580d1270 0x7f0a580d2490 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/2622437302 >> 172.21.15.161:0/2622437302 conn(0x7f0a5809d190 msgr2=0x7f0a580d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/2622437302 shutdown_connections 2024-03-23T14:35:10.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.673+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/2622437302 wait complete. 2024-03-23T14:35:10.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.674+0000 7f0a5ef6c640 1 Processor -- start 2024-03-23T14:35:10.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.674+0000 7f0a5ef6c640 1 -- start start 2024-03-23T14:35:10.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5ef6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d1270 0x7f0a581605c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5ef6c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a580d29d0 0x7f0a58160b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5ef6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a58161820 0x7f0a5809f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5ef6c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a581f4130 con 0x7f0a58161820 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5ef6c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0a580a0840 con 0x7f0a580d29d0 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5ef6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0a580cdf30 con 0x7f0a580d1270 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5d509640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a58161820 0x7f0a5809f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5d509640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a58161820 0x7f0a5809f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40862/0 (socket says 172.21.15.161:40862) 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5d509640 1 -- 172.21.15.161:0/530788568 learned_addr learned my addr 172.21.15.161:0/530788568 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a5cd08640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d1270 0x7f0a581605c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.675+0000 7f0a57fff640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a580d29d0 0x7f0a58160b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a5d509640 1 -- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d1270 msgr2=0x7f0a581605c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a5d509640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d1270 0x7f0a581605c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a5d509640 1 -- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a580d29d0 msgr2=0x7f0a58160b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a5d509640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a580d29d0 0x7f0a58160b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a5d509640 1 -- 172.21.15.161:0/530788568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a40009400 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a5d509640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a58161820 0x7f0a5809f0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800c550 tx=0x7f0a4800ca10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a557fa640 1 -- 172.21.15.161:0/530788568 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48004220 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a557fa640 1 -- 172.21.15.161:0/530788568 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a48004380 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a5809fcc0 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a557fa640 1 -- 172.21.15.161:0/530788568 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a48003830 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.676+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a580a0280 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.677+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a581f42a0 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.677+0000 7f0a557fa640 1 -- 172.21.15.161:0/530788568 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0a48003990 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.678+0000 7f0a557fa640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a3806b810 0x7f0a3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.678+0000 7f0a557fa640 1 -- 172.21.15.161:0/530788568 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7f0a48089b30 con 0x7f0a58161820 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.678+0000 7f0a5cd08640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a3806b810 0x7f0a3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.679+0000 7f0a5cd08640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a3806b810 0x7f0a3806dcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a500012e0 tx=0x7f0a50007ed0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.682+0000 7f0a557fa640 1 -- 172.21.15.161:0/530788568 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0a581f42a0 con 0x7f0a58161820 2024-03-23T14:35:10.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.860+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f0a581f4270 con 0x7f0a58161820 2024-03-23T14:35:10.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.860+0000 7f0a557fa640 1 -- 172.21.15.161:0/530788568 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v16) ==== 84+0+65 (secure 0 0 0) 0x7f0a480548b0 con 0x7f0a58161820 2024-03-23T14:35:10.856 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:10.856 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-23T14:35:10.856 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:35:10.856 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-23T14:35:10.856 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-23T14:35:10.856 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:35:10.856 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-23T14:35:10.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a3806b810 msgr2=0x7f0a3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a3806b810 0x7f0a3806dcd0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f0a500012e0 tx=0x7f0a50007ed0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a58161820 msgr2=0x7f0a5809f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a58161820 0x7f0a5809f0e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0a4800c550 tx=0x7f0a4800ca10 comp rx=0 tx=0).stop 2024-03-23T14:35:10.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 shutdown_connections 2024-03-23T14:35:10.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a3806b810 0x7f0a3806dcd0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a58161820 0x7f0a5809f0e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a580d29d0 0x7f0a58160b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 --2- 172.21.15.161:0/530788568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a580d1270 0x7f0a581605c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 >> 172.21.15.161:0/530788568 conn(0x7f0a5809d190 msgr2=0x7f0a5815feb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:10.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 shutdown_connections 2024-03-23T14:35:10.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.862+0000 7f0a5ef6c640 1 -- 172.21.15.161:0/530788568 wait complete. 2024-03-23T14:35:10.868 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T14:35:10.869 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T14:35:10.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.977+0000 7ff7d5547640 1 Processor -- start 2024-03-23T14:35:10.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.977+0000 7ff7d5547640 1 -- start start 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7d5547640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff7d00d3870 0x7ff7d0168f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7d5547640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7d00d1e10 con 0x7ff7d00cfca0 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7d5547640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7d00d4970 con 0x7ff7d015e550 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7d5547640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7ff7d00d4c50 con 0x7ff7d00d3870 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7ce7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff7d00cfca0 0x7ff7d00cf1c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49746/0 (socket says 172.21.15.161:49746) 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7ce7fc640 1 -- 172.21.15.161:0/1922572230 learned_addr learned my addr 172.21.15.161:0/1922572230 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7cf7fe640 1 --2- 172.21.15.161:0/1922572230 >> v2:172.21.15.183:3300/0 conn(0x7ff7d00d3870 0x7ff7d0168f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 654973682 0 0) 0x7ff7d00d1e10 con 0x7ff7d00cfca0 2024-03-23T14:35:10.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7b8003520 con 0x7ff7d00cfca0 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1053086119 0 0) 0x7ff7d00d4970 con 0x7ff7d015e550 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff7d00d1e10 con 0x7ff7d015e550 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2328089813 0 0) 0x7ff7b8003520 con 0x7ff7d00cfca0 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 >> v2:172.21.15.183:3300/0 conn(0x7ff7d00d3870 msgr2=0x7ff7d0168f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.978+0000 7ff7cdffb640 1 --2- 172.21.15.161:0/1922572230 >> v2:172.21.15.183:3300/0 conn(0x7ff7d00d3870 0x7ff7d0168f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 >> v1:172.21.15.183:6789/0 conn(0x7ff7d015e550 legacy=0x7ff7d00ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d00d2060 con 0x7ff7d00cfca0 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7ff7c4001670 con 0x7ff7d00cfca0 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 >> v1:172.21.15.161:6789/0 conn(0x7ff7d00cfca0 legacy=0x7ff7d00cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b8003b00 0x7ff7b8003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b80047c0 0x7ff7b8004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b8005180 0x7ff7b800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d00d4c50 con 0x7ff7b8005180 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff7b800c0b0 con 0x7ff7b8003b00 2024-03-23T14:35:10.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff7b800c270 con 0x7ff7b80047c0 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cf7fe640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b8005180 0x7ff7b800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7ceffd640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b8003b00 0x7ff7b8003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7ce7fc640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b80047c0 0x7ff7b8004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cf7fe640 1 -- 172.21.15.161:0/1922572230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b80047c0 msgr2=0x7ff7b8004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cf7fe640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b80047c0 0x7ff7b8004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cf7fe640 1 -- 172.21.15.161:0/1922572230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b8003b00 msgr2=0x7ff7b8003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cf7fe640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7b8003b00 0x7ff7b8003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.979+0000 7ff7cf7fe640 1 -- 172.21.15.161:0/1922572230 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7b8003520 con 0x7ff7b8005180 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7cf7fe640 1 -- 172.21.15.161:0/1922572230 reap_dead start 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7cf7fe640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b8005180 0x7ff7b800b980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009fc0 tx=0x7ff7b80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2265c9a7079d65db server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c802cc60 con 0x7ff7b8005180 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c800f040 con 0x7ff7b8005180 2024-03-23T14:35:10.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7cdffb640 1 -- 172.21.15.161:0/1922572230 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c8003990 con 0x7ff7b8005180 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7d5547640 1 -- 172.21.15.161:0/1922572230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b8005180 msgr2=0x7ff7b800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7d5547640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b8005180 0x7ff7b800b980 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7ff7c8009fc0 tx=0x7ff7b80057f0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7d5547640 1 -- 172.21.15.161:0/1922572230 shutdown_connections 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7d5547640 1 --2- 172.21.15.161:0/1922572230 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7b8005180 0x7ff7b800b980 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7d5547640 1 -- 172.21.15.161:0/1922572230 >> 172.21.15.161:0/1922572230 conn(0x7ff7d009d210 msgr2=0x7ff7d00d3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7d5547640 1 -- 172.21.15.161:0/1922572230 shutdown_connections 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.980+0000 7ff7d5547640 1 -- 172.21.15.161:0/1922572230 wait complete. 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7d5547640 1 Processor -- start 2024-03-23T14:35:10.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7d5547640 1 -- start start 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7d5547640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b8005180 0x7ff7d0152d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7d5547640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d015f790 0x7ff7d015fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7d5547640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7d015c5f0 0x7ff7d015ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7d5547640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d015cfb0 con 0x7ff7d015f790 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7d5547640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d00b7d20 con 0x7ff7d015c5f0 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7d5547640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d00ccef0 con 0x7ff7b8005180 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.982+0000 7ff7ce7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d015f790 0x7ff7d015fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ce7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d015f790 0x7ff7d015fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40894/0 (socket says 172.21.15.161:40894) 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ce7fc640 1 -- 172.21.15.161:0/4253553045 learned_addr learned my addr 172.21.15.161:0/4253553045 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ceffd640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b8005180 0x7ff7d0152d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7cf7fe640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7d015c5f0 0x7ff7d015ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ce7fc640 1 -- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b8005180 msgr2=0x7ff7d0152d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ce7fc640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b8005180 0x7ff7d0152d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ce7fc640 1 -- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7d015c5f0 msgr2=0x7ff7d015ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ce7fc640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7d015c5f0 0x7ff7d015ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ce7fc640 1 -- 172.21.15.161:0/4253553045 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d00b7f70 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7ce7fc640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d015f790 0x7ff7d015fc10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400e730 tx=0x7ff7c400eb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7b77fe640 1 -- 172.21.15.161:0/4253553045 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400ee70 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7b77fe640 1 -- 172.21.15.161:0/4253553045 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c400c6c0 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d00b8250 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7b77fe640 1 -- 172.21.15.161:0/4253553045 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c400ca70 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.983+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d0153240 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.984+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7d00a1f60 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.984+0000 7ff7b77fe640 1 -- 172.21.15.161:0/4253553045 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff7c400cbd0 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.985+0000 7ff7b77fe640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff7bc06b810 0x7ff7bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.985+0000 7ff7b77fe640 1 -- 172.21.15.161:0/4253553045 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(16..16 src has 1..16) ==== 4857+0+0 (secure 0 0 0) 0x7ff7c4089c50 con 0x7ff7d015f790 2024-03-23T14:35:10.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.985+0000 7ff7ceffd640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff7bc06b810 0x7ff7bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:10.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.985+0000 7ff7ceffd640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff7bc06b810 0x7ff7bc06dcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0000c00 tx=0x7ff7c0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:10.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:10.989+0000 7ff7b77fe640 1 -- 172.21.15.161:0/4253553045 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff7c408f020 con 0x7ff7d015f790 2024-03-23T14:35:11.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.161+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7ff7d004f550 con 0x7ff7d015f790 2024-03-23T14:35:11.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.744+0000 7ff7b77fe640 1 -- 172.21.15.161:0/4253553045 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v17) ==== 138+0+0 (secure 0 0 0) 0x7ff7c40549f0 con 0x7ff7d015f790 2024-03-23T14:35:11.740 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-23T14:35:11.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff7bc06b810 msgr2=0x7ff7bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:11.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff7bc06b810 0x7ff7bc06dcd0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7ff7c0000c00 tx=0x7ff7c0009040 comp rx=0 tx=0).stop 2024-03-23T14:35:11.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d015f790 msgr2=0x7ff7d015fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:11.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d015f790 0x7ff7d015fc10 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff7c400e730 tx=0x7ff7c400eb30 comp rx=0 tx=0).stop 2024-03-23T14:35:11.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 shutdown_connections 2024-03-23T14:35:11.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff7bc06b810 0x7ff7bc06dcd0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff7d015c5f0 0x7ff7d015ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff7d015f790 0x7ff7d015fc10 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 --2- 172.21.15.161:0/4253553045 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff7b8005180 0x7ff7d0152d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 >> 172.21.15.161:0/4253553045 conn(0x7ff7d009d210 msgr2=0x7ff7d00cf170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:11.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 shutdown_connections 2024-03-23T14:35:11.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.748+0000 7ff7d5547640 1 -- 172.21.15.161:0/4253553045 wait complete. 2024-03-23T14:35:11.758 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T14:35:11.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.869+0000 7f2f48aeb640 1 Processor -- start 2024-03-23T14:35:11.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.869+0000 7f2f48aeb640 1 -- start start 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.870+0000 7f2f48aeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4415bfc0 0x7f2f440ce5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.870+0000 7f2f48aeb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2f440d0200 0x7f2f440ceb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.870+0000 7f2f48aeb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f440d0c50 con 0x7f2f440cd8b0 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.870+0000 7f2f48aeb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f2f440c5c80 con 0x7f2f440d0200 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.870+0000 7f2f48aeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2f440d0690 con 0x7f2f4415bfc0 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f41d9b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2f440d0200 0x7f2f440ceb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f4259c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4415bfc0 0x7f2f440ce5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f42d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2f440cd8b0 0x7f2f440cc730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33796/0 (socket says 172.21.15.161:33796) 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f42d9d640 1 -- 172.21.15.161:0/2930086268 learned_addr learned my addr 172.21.15.161:0/2930086268 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f41d9b640 1 -- 172.21.15.161:0/2930086268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4415bfc0 msgr2=0x7f2f440ce5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f41d9b640 1 --2- 172.21.15.161:0/2930086268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4415bfc0 0x7f2f440ce5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f41d9b640 1 -- 172.21.15.161:0/2930086268 >> v1:172.21.15.183:6789/0 conn(0x7f2f440cd8b0 legacy=0x7f2f440cc730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:11.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f41d9b640 1 -- 172.21.15.161:0/2930086268 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f440d0970 con 0x7f2f440d0200 2024-03-23T14:35:11.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.871+0000 7f2f4159a640 1 -- 172.21.15.161:0/2930086268 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1738021550 0 0) 0x7f2f3c001110 con 0x7f2f440cd8b0 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.872+0000 7f2f41d9b640 1 --2- 172.21.15.161:0/2930086268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440d0200 0x7f2f440ceb30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2f380097a0 tx=0x7f2f3802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4822a435940c288e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.872+0000 7f2f4159a640 1 -- 172.21.15.161:0/2930086268 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38031830 con 0x7f2f440d0200 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.872+0000 7f2f4159a640 1 -- 172.21.15.161:0/2930086268 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f38031990 con 0x7f2f440d0200 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.872+0000 7f2f4159a640 1 -- 172.21.15.161:0/2930086268 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38031d20 con 0x7f2f440d0200 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.873+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2930086268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440d0200 msgr2=0x7f2f440ceb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.873+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2930086268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440d0200 0x7f2f440ceb30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2f380097a0 tx=0x7f2f3802b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.873+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2930086268 shutdown_connections 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.873+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2930086268 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440d0200 0x7f2f440ceb30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.873+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2930086268 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4415bfc0 0x7f2f440ce5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.873+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2930086268 >> 172.21.15.161:0/2930086268 conn(0x7f2f440b3750 msgr2=0x7f2f440b3b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:11.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.873+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2930086268 shutdown_connections 2024-03-23T14:35:11.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.873+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2930086268 wait complete. 2024-03-23T14:35:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.874+0000 7f2f48aeb640 1 Processor -- start 2024-03-23T14:35:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.874+0000 7f2f48aeb640 1 -- start start 2024-03-23T14:35:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f48aeb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440cd8b0 0x7f2f440aea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f48aeb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f4415bfc0 0x7f2f4414f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f48aeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4414dc70 0x7f2f4414f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f48aeb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4414e170 con 0x7f2f4415bfc0 2024-03-23T14:35:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f48aeb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4414e450 con 0x7f2f440cd8b0 2024-03-23T14:35:11.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f48aeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2f4414e730 con 0x7f2f4414dc70 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f4415bfc0 0x7f2f4414f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f4415bfc0 0x7f2f4414f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40908/0 (socket says 172.21.15.161:40908) 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 -- 172.21.15.161:0/2223352651 learned_addr learned my addr 172.21.15.161:0/2223352651 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f4259c640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440cd8b0 0x7f2f440aea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f42d9d640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4414dc70 0x7f2f4414f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 -- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4414dc70 msgr2=0x7f2f4414f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4414dc70 0x7f2f4414f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 -- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440cd8b0 msgr2=0x7f2f440aea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440cd8b0 0x7f2f440aea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 -- 172.21.15.161:0/2223352651 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f38009400 con 0x7f2f4415bfc0 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f4259c640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440cd8b0 0x7f2f440aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f41d9b640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f4415bfc0 0x7f2f4414f210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2f38000c00 tx=0x7f2f380046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f32ffd640 1 -- 172.21.15.161:0/2223352651 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f38004230 con 0x7f2f4415bfc0 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f32ffd640 1 -- 172.21.15.161:0/2223352651 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f38004390 con 0x7f2f4415bfc0 2024-03-23T14:35:11.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f4414ea10 con 0x7f2f4415bfc0 2024-03-23T14:35:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.875+0000 7f2f32ffd640 1 -- 172.21.15.161:0/2223352651 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f3803a8a0 con 0x7f2f4415bfc0 2024-03-23T14:35:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.876+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f440aef70 con 0x7f2f4415bfc0 2024-03-23T14:35:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.876+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4414ef30 con 0x7f2f4415bfc0 2024-03-23T14:35:11.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.877+0000 7f2f32ffd640 1 -- 172.21.15.161:0/2223352651 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2f3803aa00 con 0x7f2f4415bfc0 2024-03-23T14:35:11.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.877+0000 7f2f32ffd640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2406b840 0x7f2f2406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:11.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.877+0000 7f2f32ffd640 1 -- 172.21.15.161:0/2223352651 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 5227+0+0 (secure 0 0 0) 0x7f2f380ac540 con 0x7f2f4415bfc0 2024-03-23T14:35:11.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.878+0000 7f2f4259c640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2406b840 0x7f2f2406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:11.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.878+0000 7f2f4259c640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2406b840 0x7f2f2406dd00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f440b79e0 tx=0x7f2f3400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:11.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:11.882+0000 7f2f32ffd640 1 -- 172.21.15.161:0/2223352651 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2f4414ef30 con 0x7f2f4415bfc0 2024-03-23T14:35:12.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.055+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f2f4414ef30 con 0x7f2f4415bfc0 2024-03-23T14:35:12.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.754+0000 7f2f32ffd640 1 -- 172.21.15.161:0/2223352651 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v18) ==== 170+0+0 (secure 0 0 0) 0x7f2f4414ef30 con 0x7f2f4415bfc0 2024-03-23T14:35:12.750 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-23T14:35:12.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.758+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2406b840 msgr2=0x7f2f2406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.758+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2406b840 0x7f2f2406dd00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f440b79e0 tx=0x7f2f3400b040 comp rx=0 tx=0).stop 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.758+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f4415bfc0 msgr2=0x7f2f4414f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.758+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f4415bfc0 0x7f2f4414f210 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f2f38000c00 tx=0x7f2f380046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.759+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 shutdown_connections 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.759+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2406b840 0x7f2f2406dd00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.759+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f4414dc70 0x7f2f4414f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.759+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f4415bfc0 0x7f2f4414f210 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.759+0000 7f2f48aeb640 1 --2- 172.21.15.161:0/2223352651 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f440cd8b0 0x7f2f440aea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.759+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 >> 172.21.15.161:0/2223352651 conn(0x7f2f440b3750 msgr2=0x7f2f440c7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:12.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.759+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 shutdown_connections 2024-03-23T14:35:12.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.759+0000 7f2f48aeb640 1 -- 172.21.15.161:0/2223352651 wait complete. 2024-03-23T14:35:12.778 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T14:35:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.890+0000 7f96ee302640 1 Processor -- start 2024-03-23T14:35:12.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.890+0000 7f96ee302640 1 -- start start 2024-03-23T14:35:12.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96ee302640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f96e80b43f0 0x7f96e815b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:12.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96ee302640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e815b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:12.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96ee302640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e815c1f0 con 0x7f96e80b4010 2024-03-23T14:35:12.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96ee302640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f96e806ef20 con 0x7f96e80b43f0 2024-03-23T14:35:12.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96ee302640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f96e815f730 con 0x7f96e80d05f0 2024-03-23T14:35:12.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96ec89f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e815b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:12.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96e77fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f96e80b43f0 0x7f96e815b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:12.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96e7fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f96e80b4010 0x7f96e80cfee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33800/0 (socket says 172.21.15.161:33800) 2024-03-23T14:35:12.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.891+0000 7f96ec89f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e815b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:41992/0 (socket says 172.21.15.161:41992) 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.892+0000 7f96e7fff640 1 -- 172.21.15.161:0/474625970 learned_addr learned my addr 172.21.15.161:0/474625970 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.892+0000 7f96ec89f640 1 -- 172.21.15.161:0/474625970 >> v2:172.21.15.183:3300/0 conn(0x7f96e80b43f0 msgr2=0x7f96e815b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.892+0000 7f96ec89f640 1 --2- 172.21.15.161:0/474625970 >> v2:172.21.15.183:3300/0 conn(0x7f96e80b43f0 0x7f96e815b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.892+0000 7f96ec89f640 1 -- 172.21.15.161:0/474625970 >> v1:172.21.15.183:6789/0 conn(0x7f96e80b4010 legacy=0x7f96e80cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.892+0000 7f96ec89f640 1 -- 172.21.15.161:0/474625970 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e815e970 con 0x7f96e80d05f0 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ec89f640 1 --2- 172.21.15.161:0/474625970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e815b9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009dc0 tx=0x7f96e002be00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0e85717e7bce588 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96e6ffd640 1 -- 172.21.15.161:0/474625970 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e002cbb0 con 0x7f96e80d05f0 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96e6ffd640 1 -- 172.21.15.161:0/474625970 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96e002cd10 con 0x7f96e80d05f0 2024-03-23T14:35:12.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96e6ffd640 1 -- 172.21.15.161:0/474625970 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e0034660 con 0x7f96e80d05f0 2024-03-23T14:35:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ee302640 1 -- 172.21.15.161:0/474625970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 msgr2=0x7f96e815b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ee302640 1 --2- 172.21.15.161:0/474625970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e815b9c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f96e0009dc0 tx=0x7f96e002be00 comp rx=0 tx=0).stop 2024-03-23T14:35:12.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ee302640 1 -- 172.21.15.161:0/474625970 shutdown_connections 2024-03-23T14:35:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ee302640 1 --2- 172.21.15.161:0/474625970 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e815b9c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ee302640 1 --2- 172.21.15.161:0/474625970 >> v2:172.21.15.183:3300/0 conn(0x7f96e80b43f0 0x7f96e815b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ee302640 1 -- 172.21.15.161:0/474625970 >> 172.21.15.161:0/474625970 conn(0x7f96e806b4a0 msgr2=0x7f96e806b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ee302640 1 -- 172.21.15.161:0/474625970 shutdown_connections 2024-03-23T14:35:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.893+0000 7f96ee302640 1 -- 172.21.15.161:0/474625970 wait complete. 2024-03-23T14:35:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.894+0000 7f96ee302640 1 Processor -- start 2024-03-23T14:35:12.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96ee302640 1 -- start start 2024-03-23T14:35:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96ee302640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e80b4010 0x7f96e8071c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96ee302640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e80b43f0 0x7f96e8070c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96ee302640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e814c090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96ee302640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96e80ada30 con 0x7f96e80b4010 2024-03-23T14:35:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96ee302640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f96e80add10 con 0x7f96e80b43f0 2024-03-23T14:35:12.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96ee302640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f96e80adff0 con 0x7f96e80d05f0 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96e7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e80b4010 0x7f96e8071c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96e7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e80b4010 0x7f96e8071c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40912/0 (socket says 172.21.15.161:40912) 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96e7fff640 1 -- 172.21.15.161:0/2541616208 learned_addr learned my addr 172.21.15.161:0/2541616208 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96e77fe640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e80b43f0 0x7f96e8070c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.895+0000 7f96ec89f640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e814c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e7fff640 1 -- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 msgr2=0x7f96e814c090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e7fff640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e814c090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e7fff640 1 -- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e80b43f0 msgr2=0x7f96e8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e7fff640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e80b43f0 0x7f96e8070c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e7fff640 1 -- 172.21.15.161:0/2541616208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e0009a70 con 0x7f96e80b4010 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e7fff640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e80b4010 0x7f96e8071c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f96d800c400 tx=0x7f96d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e4ff9640 1 -- 172.21.15.161:0/2541616208 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d8005ce0 con 0x7f96e80b4010 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e4ff9640 1 -- 172.21.15.161:0/2541616208 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d8005e40 con 0x7f96e80b4010 2024-03-23T14:35:12.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96e80ae2d0 con 0x7f96e80b4010 2024-03-23T14:35:12.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96e4ff9640 1 -- 172.21.15.161:0/2541616208 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d80072d0 con 0x7f96e80b4010 2024-03-23T14:35:12.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.896+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e8153bf0 con 0x7f96e80b4010 2024-03-23T14:35:12.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.897+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e8073570 con 0x7f96e80b4010 2024-03-23T14:35:12.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.898+0000 7f96e4ff9640 1 -- 172.21.15.161:0/2541616208 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f96d80074f0 con 0x7f96e80b4010 2024-03-23T14:35:12.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.898+0000 7f96e4ff9640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96d406b810 0x7f96d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:12.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.898+0000 7f96e4ff9640 1 -- 172.21.15.161:0/2541616208 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5609+0+0 (secure 0 0 0) 0x7f96d80896e0 con 0x7f96e80b4010 2024-03-23T14:35:12.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.898+0000 7f96e77fe640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96d406b810 0x7f96d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:12.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.899+0000 7f96e77fe640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96d406b810 0x7f96d406dcd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f96e80b2e30 tx=0x7f96dc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:12.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:12.903+0000 7f96e4ff9640 1 -- 172.21.15.161:0/2541616208 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f96d8054210 con 0x7f96e80b4010 2024-03-23T14:35:13.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.110+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f96e80b31d0 con 0x7f96e80b4010 2024-03-23T14:35:13.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.755+0000 7f96e4ff9640 1 -- 172.21.15.161:0/2541616208 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7f96d80578d0 con 0x7f96e80b4010 2024-03-23T14:35:13.751 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 2 and data pool 3 2024-03-23T14:35:13.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96d406b810 msgr2=0x7f96d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96d406b810 0x7f96d406dcd0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f96e80b2e30 tx=0x7f96dc009190 comp rx=0 tx=0).stop 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e80b4010 msgr2=0x7f96e8071c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e80b4010 0x7f96e8071c40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f96d800c400 tx=0x7f96d800c8c0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 shutdown_connections 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96d406b810 0x7f96d406dcd0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e80d05f0 0x7f96e814c090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e80b43f0 0x7f96e8070c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 --2- 172.21.15.161:0/2541616208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e80b4010 0x7f96e8071c40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 >> 172.21.15.161:0/2541616208 conn(0x7f96e806b4a0 msgr2=0x7f96e80b0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 shutdown_connections 2024-03-23T14:35:13.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.758+0000 7f96ee302640 1 -- 172.21.15.161:0/2541616208 wait complete. 2024-03-23T14:35:13.777 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:13.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.884+0000 7f36937e2640 1 Processor -- start 2024-03-23T14:35:13.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.884+0000 7f36937e2640 1 -- start start 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.885+0000 7f36937e2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f368c15e060 0x7f368c0727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.885+0000 7f36937e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0d14c0 0x7f368c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.885+0000 7f36937e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f368c15e440 con 0x7f368c0b5b80 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.885+0000 7f36937e2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f368c0b0480 con 0x7f368c15e060 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.885+0000 7f36937e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f368c0b5fe0 con 0x7f368c0d14c0 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.885+0000 7f3691d7f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0d14c0 0x7f368c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.885+0000 7f369157e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f368c15e060 0x7f368c0727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f3690d7d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f368c0b5b80 0x7f368c0d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33808/0 (socket says 172.21.15.161:33808) 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f3691d7f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0d14c0 0x7f368c06dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42014/0 (socket says 172.21.15.161:42014) 2024-03-23T14:35:13.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f3690d7d640 1 -- 172.21.15.161:0/2780229717 learned_addr learned my addr 172.21.15.161:0/2780229717 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f3691d7f640 1 -- 172.21.15.161:0/2780229717 >> v2:172.21.15.183:3300/0 conn(0x7f368c15e060 msgr2=0x7f368c0727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f3691d7f640 1 --2- 172.21.15.161:0/2780229717 >> v2:172.21.15.183:3300/0 conn(0x7f368c15e060 0x7f368c0727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f3691d7f640 1 -- 172.21.15.161:0/2780229717 >> v1:172.21.15.183:6789/0 conn(0x7f368c0b5b80 legacy=0x7f368c0d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f3683fff640 1 -- 172.21.15.161:0/2780229717 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2998646345 0 0) 0x7f368c15e440 con 0x7f368c0b5b80 2024-03-23T14:35:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f3691d7f640 1 -- 172.21.15.161:0/2780229717 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368c161cf0 con 0x7f368c0d14c0 2024-03-23T14:35:13.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.886+0000 7f369157e640 1 --2- 172.21.15.161:0/2780229717 >> v2:172.21.15.183:3300/0 conn(0x7f368c15e060 0x7f368c0727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:13.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f3691d7f640 1 --2- 172.21.15.161:0/2780229717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0d14c0 0x7f368c06dcd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f36840097a0 tx=0x7f368402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a56117e8fd4fdb2e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f3683fff640 1 -- 172.21.15.161:0/2780229717 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3684031830 con 0x7f368c0d14c0 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f3683fff640 1 -- 172.21.15.161:0/2780229717 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3684031990 con 0x7f368c0d14c0 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f3683fff640 1 -- 172.21.15.161:0/2780229717 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3684031d20 con 0x7f368c0d14c0 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f36937e2640 1 -- 172.21.15.161:0/2780229717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0d14c0 msgr2=0x7f368c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f36937e2640 1 --2- 172.21.15.161:0/2780229717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0d14c0 0x7f368c06dcd0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f36840097a0 tx=0x7f368402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f36937e2640 1 -- 172.21.15.161:0/2780229717 shutdown_connections 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f36937e2640 1 --2- 172.21.15.161:0/2780229717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0d14c0 0x7f368c06dcd0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f36937e2640 1 --2- 172.21.15.161:0/2780229717 >> v2:172.21.15.183:3300/0 conn(0x7f368c15e060 0x7f368c0727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f36937e2640 1 -- 172.21.15.161:0/2780229717 >> 172.21.15.161:0/2780229717 conn(0x7f368c06c370 msgr2=0x7f368c06c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f36937e2640 1 -- 172.21.15.161:0/2780229717 shutdown_connections 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.887+0000 7f36937e2640 1 -- 172.21.15.161:0/2780229717 wait complete. 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f36937e2640 1 Processor -- start 2024-03-23T14:35:13.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f36937e2640 1 -- start start 2024-03-23T14:35:13.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f36937e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0b5b80 0x7f368c1450c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:13.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f36937e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0d14c0 0x7f368c160ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:13.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f36937e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c15e060 0x7f368c1f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:13.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f36937e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f368c161430 con 0x7f368c0d14c0 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f36937e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f368c1f3ec0 con 0x7f368c15e060 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f36937e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f368c1f41a0 con 0x7f368c0b5b80 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f3690d7d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0d14c0 0x7f368c160ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f3690d7d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0d14c0 0x7f368c160ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40926/0 (socket says 172.21.15.161:40926) 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f3690d7d640 1 -- 172.21.15.161:0/3818939986 learned_addr learned my addr 172.21.15.161:0/3818939986 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f369157e640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0b5b80 0x7f368c1450c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.889+0000 7f3691d7f640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c15e060 0x7f368c1f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3690d7d640 1 -- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0b5b80 msgr2=0x7f368c1450c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3690d7d640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0b5b80 0x7f368c1450c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3690d7d640 1 -- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c15e060 msgr2=0x7f368c1f3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3690d7d640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c15e060 0x7f368c1f3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3690d7d640 1 -- 172.21.15.161:0/3818939986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3688009a70 con 0x7f368c0d14c0 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3690d7d640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0d14c0 0x7f368c160ef0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c4a0 tx=0x7f367c00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:13.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3681ffb640 1 -- 172.21.15.161:0/3818939986 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c0074e0 con 0x7f368c0d14c0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3681ffb640 1 -- 172.21.15.161:0/3818939986 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f367c007640 con 0x7f368c0d14c0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f3681ffb640 1 -- 172.21.15.161:0/3818939986 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f367c004a00 con 0x7f368c0d14c0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3684009400 con 0x7f368c0d14c0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.890+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f368c0b1ca0 con 0x7f368c0d14c0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.891+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f368c0b1e90 con 0x7f368c0d14c0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.892+0000 7f3681ffb640 1 -- 172.21.15.161:0/3818939986 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f367c004b60 con 0x7f368c0d14c0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.892+0000 7f3681ffb640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f367006b810 0x7f367006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.892+0000 7f3681ffb640 1 -- 172.21.15.161:0/3818939986 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f367c089280 con 0x7f368c0d14c0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.892+0000 7f369157e640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f367006b810 0x7f367006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:13.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.893+0000 7f369157e640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f367006b810 0x7f367006dcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f36880114e0 tx=0x7f368800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:13.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:13.896+0000 7f3681ffb640 1 -- 172.21.15.161:0/3818939986 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f367c053d60 con 0x7f368c0d14c0 2024-03-23T14:35:14.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.069+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f368c073680 con 0x7f368c0d14c0 2024-03-23T14:35:14.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.070+0000 7f3681ffb640 1 -- 172.21.15.161:0/3818939986 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16654 (secure 0 0 0) 0x7f367c057420 con 0x7f368c0d14c0 2024-03-23T14:35:14.066 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:14.067 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":19,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:13.112139+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:14.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f367006b810 msgr2=0x7f367006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f367006b810 0x7f367006dcd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f36880114e0 tx=0x7f368800b040 comp rx=0 tx=0).stop 2024-03-23T14:35:14.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0d14c0 msgr2=0x7f368c160ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0d14c0 0x7f368c160ef0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f367c00c4a0 tx=0x7f367c00c960 comp rx=0 tx=0).stop 2024-03-23T14:35:14.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 shutdown_connections 2024-03-23T14:35:14.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f367006b810 0x7f367006dcd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f368c15e060 0x7f368c1f3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f368c0d14c0 0x7f368c160ef0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 --2- 172.21.15.161:0/3818939986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f368c0b5b80 0x7f368c1450c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 >> 172.21.15.161:0/3818939986 conn(0x7f368c06c370 msgr2=0x7f368c0d0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:14.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 shutdown_connections 2024-03-23T14:35:14.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.072+0000 7f36937e2640 1 -- 172.21.15.161:0/3818939986 wait complete. 2024-03-23T14:35:14.079 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:14.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.183+0000 7fd9cff6c640 1 Processor -- start 2024-03-23T14:35:14.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.184+0000 7fd9cff6c640 1 -- start start 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.184+0000 7fd9cff6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c815e400 0x7fd9c80cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.184+0000 7fd9cff6c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c80d0460 con 0x7fd9c80cf9d0 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.184+0000 7fd9cff6c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c80d06b0 con 0x7fd9c80d2b40 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.184+0000 7fd9cff6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c80d1050 con 0x7fd9c815e400 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9cd507640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd9c80cf9d0 0x7fd9c80d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49758/0 (socket says 172.21.15.161:49758) 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9cd507640 1 -- 172.21.15.161:0/3289668501 learned_addr learned my addr 172.21.15.161:0/3289668501 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9cdd08640 1 --2- 172.21.15.161:0/3289668501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c815e400 0x7fd9c80cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9ccd06640 1 -- 172.21.15.161:0/3289668501 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 174709446 0 0) 0x7fd9c80d0460 con 0x7fd9c80cf9d0 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9ccd06640 1 -- 172.21.15.161:0/3289668501 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b4003520 con 0x7fd9c80cf9d0 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9ccd06640 1 -- 172.21.15.161:0/3289668501 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 194354248 0 0) 0x7fd9c80d06b0 con 0x7fd9c80d2b40 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9ccd06640 1 -- 172.21.15.161:0/3289668501 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c80d0460 con 0x7fd9c80d2b40 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9cdd08640 1 -- 172.21.15.161:0/3289668501 >> v1:172.21.15.183:6789/0 conn(0x7fd9c80d2b40 legacy=0x7fd9c80b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9cdd08640 1 -- 172.21.15.161:0/3289668501 >> v1:172.21.15.161:6789/0 conn(0x7fd9c80cf9d0 legacy=0x7fd9c80d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.185+0000 7fd9cdd08640 1 -- 172.21.15.161:0/3289668501 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c80d0990 con 0x7fd9c815e400 2024-03-23T14:35:14.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.186+0000 7fd9cdd08640 1 --2- 172.21.15.161:0/3289668501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c815e400 0x7fd9c80cf490 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40097d0 tx=0x7fd9c402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b8737390e585654e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.186+0000 7fd9ccd06640 1 -- 172.21.15.161:0/3289668501 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4031890 con 0x7fd9c815e400 2024-03-23T14:35:14.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.186+0000 7fd9ccd06640 1 -- 172.21.15.161:0/3289668501 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9c40319f0 con 0x7fd9c815e400 2024-03-23T14:35:14.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.186+0000 7fd9ccd06640 1 -- 172.21.15.161:0/3289668501 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9c4031d60 con 0x7fd9c815e400 2024-03-23T14:35:14.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.186+0000 7fd9cff6c640 1 -- 172.21.15.161:0/3289668501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c815e400 msgr2=0x7fd9c80cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.186+0000 7fd9cff6c640 1 --2- 172.21.15.161:0/3289668501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c815e400 0x7fd9c80cf490 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fd9c40097d0 tx=0x7fd9c402b930 comp rx=0 tx=0).stop 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.187+0000 7fd9cff6c640 1 -- 172.21.15.161:0/3289668501 shutdown_connections 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.187+0000 7fd9cff6c640 1 --2- 172.21.15.161:0/3289668501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c815e400 0x7fd9c80cf490 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.187+0000 7fd9cff6c640 1 -- 172.21.15.161:0/3289668501 >> 172.21.15.161:0/3289668501 conn(0x7fd9c809d190 msgr2=0x7fd9c80d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.187+0000 7fd9cff6c640 1 -- 172.21.15.161:0/3289668501 shutdown_connections 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.187+0000 7fd9cff6c640 1 -- 172.21.15.161:0/3289668501 wait complete. 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cff6c640 1 Processor -- start 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cff6c640 1 -- start start 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cff6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c80cf9d0 0x7fd9c80caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cff6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9c80d2b40 0x7fd9c815ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cff6c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9c815e400 0x7fd9c80cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cff6c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c81f7330 con 0x7fd9c80d2b40 2024-03-23T14:35:14.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cff6c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd9c815cfd0 con 0x7fd9c815e400 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cff6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd9c81f7540 con 0x7fd9c80cf9d0 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.188+0000 7fd9cd507640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9c80d2b40 0x7fd9c815ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cd507640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9c80d2b40 0x7fd9c815ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40934/0 (socket says 172.21.15.161:40934) 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cd507640 1 -- 172.21.15.161:0/1268165764 learned_addr learned my addr 172.21.15.161:0/1268165764 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9ce509640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9c815e400 0x7fd9c80cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cdd08640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c80cf9d0 0x7fd9c80caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cd507640 1 -- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c80cf9d0 msgr2=0x7fd9c80caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cd507640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c80cf9d0 0x7fd9c80caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cd507640 1 -- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9c815e400 msgr2=0x7fd9c80cc710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cd507640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9c815e400 0x7fd9c80cc710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cd507640 1 -- 172.21.15.161:0/1268165764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c4009400 con 0x7fd9c80d2b40 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cd507640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9c80d2b40 0x7fd9c815ca90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c530 tx=0x7fd9b800c9f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9be7fc640 1 -- 172.21.15.161:0/1268165764 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8005ce0 con 0x7fd9c80d2b40 2024-03-23T14:35:14.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9be7fc640 1 -- 172.21.15.161:0/1268165764 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9b8005e40 con 0x7fd9c80d2b40 2024-03-23T14:35:14.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c81f7820 con 0x7fd9c80d2b40 2024-03-23T14:35:14.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9be7fc640 1 -- 172.21.15.161:0/1268165764 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9b8007270 con 0x7fd9c80d2b40 2024-03-23T14:35:14.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.189+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9c81f7de0 con 0x7fd9c80d2b40 2024-03-23T14:35:14.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.190+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9c81f3db0 con 0x7fd9c80d2b40 2024-03-23T14:35:14.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.191+0000 7fd9be7fc640 1 -- 172.21.15.161:0/1268165764 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd9b80073d0 con 0x7fd9c80d2b40 2024-03-23T14:35:14.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.191+0000 7fd9be7fc640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd9ac06b810 0x7fd9ac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.191+0000 7fd9be7fc640 1 -- 172.21.15.161:0/1268165764 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7fd9b8089660 con 0x7fd9c80d2b40 2024-03-23T14:35:14.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.191+0000 7fd9cdd08640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd9ac06b810 0x7fd9ac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.191+0000 7fd9cdd08640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd9ac06b810 0x7fd9ac06dcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4036000 tx=0x7fd9c403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.195+0000 7fd9be7fc640 1 -- 172.21.15.161:0/1268165764 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd9b8054140 con 0x7fd9c80d2b40 2024-03-23T14:35:14.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.368+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd9c806b280 con 0x7fd9c80d2b40 2024-03-23T14:35:14.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.369+0000 7fd9be7fc640 1 -- 172.21.15.161:0/1268165764 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+16654 (secure 0 0 0) 0x7fd9b805ce60 con 0x7fd9c80d2b40 2024-03-23T14:35:14.365 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:14.366 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":19,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:13.112139+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd9ac06b810 msgr2=0x7fd9ac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd9ac06b810 0x7fd9ac06dcd0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd9c4036000 tx=0x7fd9c403c040 comp rx=0 tx=0).stop 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9c80d2b40 msgr2=0x7fd9c815ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9c80d2b40 0x7fd9c815ca90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd9b800c530 tx=0x7fd9b800c9f0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 shutdown_connections 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd9ac06b810 0x7fd9ac06dcd0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9c815e400 0x7fd9c80cc710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd9c80d2b40 0x7fd9c815ca90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 --2- 172.21.15.161:0/1268165764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd9c80cf9d0 0x7fd9c80caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 >> 172.21.15.161:0/1268165764 conn(0x7fd9c809d190 msgr2=0x7fd9c80d3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 shutdown_connections 2024-03-23T14:35:14.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.371+0000 7fd9cff6c640 1 -- 172.21.15.161:0/1268165764 wait complete. 2024-03-23T14:35:14.378 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T14:35:14.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.485+0000 7f46d6c81640 1 Processor -- start 2024-03-23T14:35:14.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.485+0000 7f46d6c81640 1 -- start start 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46d6c81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015de70 0x7f46d015e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46d6c81640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f46d00d2130 0x7f46d006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46d6c81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d00b3710 con 0x7f46d015e790 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46d6c81640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f46d015ecd0 con 0x7f46d00d2130 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46d6c81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f46d015efb0 con 0x7f46d015de70 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f46d015e790 0x7f46d00d1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49766/0 (socket says 172.21.15.161:49766) 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cffff640 1 -- 172.21.15.161:0/2716268978 learned_addr learned my addr 172.21.15.161:0/2716268978 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729137212 0 0) 0x7f46d00b3710 con 0x7f46d015e790 2024-03-23T14:35:14.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46d521e640 1 --2- 172.21.15.161:0/2716268978 >> v2:172.21.15.183:3300/0 conn(0x7f46d00d2130 0x7f46d006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46d4a1d640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015de70 0x7f46d015e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46c0003520 con 0x7f46d015e790 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3939351447 0 0) 0x7f46c0003520 con 0x7f46d015e790 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015de70 msgr2=0x7f46d015e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015de70 0x7f46d015e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 >> v2:172.21.15.183:3300/0 conn(0x7f46d00d2130 msgr2=0x7f46d006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/2716268978 >> v2:172.21.15.183:3300/0 conn(0x7f46d00d2130 0x7f46d006c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d015f290 con 0x7f46d015e790 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f46bc001670 con 0x7f46d015e790 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 >> v1:172.21.15.161:6789/0 conn(0x7f46d015e790 legacy=0x7f46d00d1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.486+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46c0003970 0x7f46c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46c00047c0 0x7f46c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015de70 0x7f46c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46d015ecd0 con 0x7f46c0003970 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f46d015efb0 con 0x7f46c00047c0 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f46c000b980 con 0x7f46d015de70 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46d4a1d640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46c0003970 0x7f46c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46cffff640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46c00047c0 0x7f46c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46d521e640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015de70 0x7f46c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46d4a1d640 1 -- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015de70 msgr2=0x7f46c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46d4a1d640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015de70 0x7f46c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46d4a1d640 1 -- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46c00047c0 msgr2=0x7f46c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46d4a1d640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46c00047c0 0x7f46c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.487+0000 7f46d4a1d640 1 -- 172.21.15.161:0/2716268978 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c0003520 con 0x7f46c0003970 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46d4a1d640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46c0003970 0x7f46c0003d90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009ac0 tx=0x7f46c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ba125e3731b9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c402c6b0 con 0x7f46c0003970 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c402c810 con 0x7f46c0003970 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46cf7fe640 1 -- 172.21.15.161:0/2716268978 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c402cba0 con 0x7f46c0003970 2024-03-23T14:35:14.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46d6c81640 1 -- 172.21.15.161:0/2716268978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46c0003970 msgr2=0x7f46c0003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46d6c81640 1 --2- 172.21.15.161:0/2716268978 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46c0003970 0x7f46c0003d90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f46c4009ac0 tx=0x7f46c00057f0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46d521e640 1 -- 172.21.15.161:0/2716268978 reap_dead start 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46d6c81640 1 -- 172.21.15.161:0/2716268978 shutdown_connections 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46d6c81640 1 -- 172.21.15.161:0/2716268978 >> 172.21.15.161:0/2716268978 conn(0x7f46d009aa40 msgr2=0x7f46d009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46d6c81640 1 -- 172.21.15.161:0/2716268978 shutdown_connections 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.488+0000 7f46d6c81640 1 -- 172.21.15.161:0/2716268978 wait complete. 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.489+0000 7f46d6c81640 1 Processor -- start 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.489+0000 7f46d6c81640 1 -- start start 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d6c81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46c0003970 0x7f46d00c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d6c81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00ca920 0x7f46d00cada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d6c81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00cb2e0 0x7f46d015fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d6c81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46d00af830 con 0x7f46d00cb2e0 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d6c81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f46d00afb10 con 0x7f46d00ca920 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d6c81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f46d00afdf0 con 0x7f46c0003970 2024-03-23T14:35:14.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d521e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00cb2e0 0x7f46d015fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d521e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00cb2e0 0x7f46d015fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40954/0 (socket says 172.21.15.161:40954) 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d521e640 1 -- 172.21.15.161:0/2149258259 learned_addr learned my addr 172.21.15.161:0/2149258259 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46cffff640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00ca920 0x7f46d00cada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.490+0000 7f46d4a1d640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46c0003970 0x7f46d00c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46d521e640 1 -- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46c0003970 msgr2=0x7f46d00c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46d521e640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46c0003970 0x7f46d00c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46d521e640 1 -- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00ca920 msgr2=0x7f46d00cada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46d521e640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00ca920 0x7f46d00cada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46d521e640 1 -- 172.21.15.161:0/2149258259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d00b00d0 con 0x7f46d00cb2e0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46d521e640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00cb2e0 0x7f46d015fda0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f46d0068e30 tx=0x7f46c800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46cd7fa640 1 -- 172.21.15.161:0/2149258259 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c800e040 con 0x7f46d00cb2e0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46cd7fa640 1 -- 172.21.15.161:0/2149258259 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c800f040 con 0x7f46d00cb2e0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46cd7fa640 1 -- 172.21.15.161:0/2149258259 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c8015670 con 0x7f46d00cb2e0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d00b03b0 con 0x7f46d00cb2e0 2024-03-23T14:35:14.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.491+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d00af480 con 0x7f46d00cb2e0 2024-03-23T14:35:14.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.492+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d00b05a0 con 0x7f46d00cb2e0 2024-03-23T14:35:14.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.492+0000 7f46cd7fa640 1 -- 172.21.15.161:0/2149258259 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f46c80157d0 con 0x7f46d00cb2e0 2024-03-23T14:35:14.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.493+0000 7f46cd7fa640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46b406b810 0x7f46b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.493+0000 7f46cd7fa640 1 -- 172.21.15.161:0/2149258259 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5709+0+0 (secure 0 0 0) 0x7f46c808a7c0 con 0x7f46d00cb2e0 2024-03-23T14:35:14.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.493+0000 7f46d4a1d640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46b406b810 0x7f46b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.493+0000 7f46d4a1d640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46b406b810 0x7f46b406dcd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f46c4036000 tx=0x7f46c4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.497+0000 7f46cd7fa640 1 -- 172.21.15.161:0/2149258259 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f46c8055290 con 0x7f46d00cb2e0 2024-03-23T14:35:14.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.704+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f46d009ff90 con 0x7f46d00cb2e0 2024-03-23T14:35:14.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.768+0000 7f46cd7fa640 1 -- 172.21.15.161:0/2149258259 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7f46c8058950 con 0x7f46d00cb2e0 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.771+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46b406b810 msgr2=0x7f46b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.771+0000 7f46d6c81640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46b406b810 0x7f46b406dcd0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f46c4036000 tx=0x7f46c4030000 comp rx=0 tx=0).stop 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.771+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00cb2e0 msgr2=0x7f46d015fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.771+0000 7f46d6c81640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00cb2e0 0x7f46d015fda0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f46d0068e30 tx=0x7f46c800f580 comp rx=0 tx=0).stop 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.772+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 shutdown_connections 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.772+0000 7f46d6c81640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46b406b810 0x7f46b406dcd0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.772+0000 7f46d6c81640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00cb2e0 0x7f46d015fda0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.772+0000 7f46d6c81640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00ca920 0x7f46d00cada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.772+0000 7f46d6c81640 1 --2- 172.21.15.161:0/2149258259 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46c0003970 0x7f46d00c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.772+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 >> 172.21.15.161:0/2149258259 conn(0x7f46d009aa40 msgr2=0x7f46d00cce40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.772+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 shutdown_connections 2024-03-23T14:35:14.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.772+0000 7f46d6c81640 1 -- 172.21.15.161:0/2149258259 wait complete. 2024-03-23T14:35:14.779 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T14:35:14.779 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T14:35:14.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.885+0000 7f46d64da640 1 Processor -- start 2024-03-23T14:35:14.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.885+0000 7f46d64da640 1 -- start start 2024-03-23T14:35:14.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46d64da640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d00b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46d64da640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d00b2c20 con 0x7f46d00cfee0 2024-03-23T14:35:14.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46d64da640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46d00b4070 con 0x7f46d00b3c10 2024-03-23T14:35:14.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46d64da640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f46d00b37f0 con 0x7f46d015e1d0 2024-03-23T14:35:14.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46cffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f46d00cfee0 0x7f46d006d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49782/0 (socket says 172.21.15.161:49782) 2024-03-23T14:35:14.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46cffff640 1 -- 172.21.15.161:0/3774040994 learned_addr learned my addr 172.21.15.161:0/3774040994 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:14.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/3774040994 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d00b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46ceffd640 1 -- 172.21.15.161:0/3774040994 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3982881506 0 0) 0x7f46d00b2c20 con 0x7f46d00cfee0 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46ceffd640 1 -- 172.21.15.161:0/3774040994 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46b8003520 con 0x7f46d00cfee0 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46ceffd640 1 -- 172.21.15.161:0/3774040994 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1982084872 0 0) 0x7f46d00b4070 con 0x7f46d00b3c10 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46ceffd640 1 -- 172.21.15.161:0/3774040994 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46d00b2c20 con 0x7f46d00b3c10 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46cf7fe640 1 -- 172.21.15.161:0/3774040994 >> v1:172.21.15.183:6789/0 conn(0x7f46d00b3c10 legacy=0x7f46d00d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46cf7fe640 1 -- 172.21.15.161:0/3774040994 >> v1:172.21.15.161:6789/0 conn(0x7f46d00cfee0 legacy=0x7f46d006d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.886+0000 7f46cf7fe640 1 -- 172.21.15.161:0/3774040994 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d00b2e50 con 0x7f46d015e1d0 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.887+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/3774040994 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d00b4570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f46c40097d0 tx=0x7f46c402b9c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c71334885f722400 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.887+0000 7f46ceffd640 1 -- 172.21.15.161:0/3774040994 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4030880 con 0x7f46d015e1d0 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.887+0000 7f46ceffd640 1 -- 172.21.15.161:0/3774040994 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c40309e0 con 0x7f46d015e1d0 2024-03-23T14:35:14.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.887+0000 7f46ceffd640 1 -- 172.21.15.161:0/3774040994 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c4030d50 con 0x7f46d015e1d0 2024-03-23T14:35:14.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.888+0000 7f46d64da640 1 -- 172.21.15.161:0/3774040994 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 msgr2=0x7f46d00b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.888+0000 7f46d64da640 1 --2- 172.21.15.161:0/3774040994 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d00b4570 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f46c40097d0 tx=0x7f46c402b9c0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.888+0000 7f46d64da640 1 -- 172.21.15.161:0/3774040994 shutdown_connections 2024-03-23T14:35:14.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.888+0000 7f46d64da640 1 --2- 172.21.15.161:0/3774040994 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d00b4570 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.888+0000 7f46d64da640 1 -- 172.21.15.161:0/3774040994 >> 172.21.15.161:0/3774040994 conn(0x7f46d006b4a0 msgr2=0x7f46d006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.888+0000 7f46d64da640 1 -- 172.21.15.161:0/3774040994 shutdown_connections 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.888+0000 7f46d64da640 1 -- 172.21.15.161:0/3774040994 wait complete. 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.889+0000 7f46d64da640 1 Processor -- start 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.889+0000 7f46d64da640 1 -- start start 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46d64da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00b3c10 0x7f46d006ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46d64da640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00cfee0 0x7f46d0147c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46d64da640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d01f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46d64da640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46d0145bd0 con 0x7f46d00b3c10 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46d64da640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f46d01486e0 con 0x7f46d00cfee0 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46d64da640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f46d01489c0 con 0x7f46d015e1d0 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00b3c10 0x7f46d006ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00b3c10 0x7f46d006ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40964/0 (socket says 172.21.15.161:40964) 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cffff640 1 -- 172.21.15.161:0/341338896 learned_addr learned my addr 172.21.15.161:0/341338896 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:14.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00cfee0 0x7f46d0147c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46d4a77640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d01f2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cffff640 1 -- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 msgr2=0x7f46d01f2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cffff640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d01f2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cffff640 1 -- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00cfee0 msgr2=0x7f46d0147c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cffff640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00cfee0 0x7f46d0147c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.890+0000 7f46cffff640 1 -- 172.21.15.161:0/341338896 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46c4009400 con 0x7f46d00b3c10 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.891+0000 7f46cffff640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00b3c10 0x7f46d006ce40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c500 tx=0x7f46c000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.891+0000 7f46ccff9640 1 -- 172.21.15.161:0/341338896 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c0005ce0 con 0x7f46d00b3c10 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.891+0000 7f46ccff9640 1 -- 172.21.15.161:0/341338896 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f46c0005e40 con 0x7f46d00b3c10 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.891+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d0145e20 con 0x7f46d00b3c10 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.891+0000 7f46ccff9640 1 -- 172.21.15.161:0/341338896 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46c00072f0 con 0x7f46d00b3c10 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.891+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d014be90 con 0x7f46d00b3c10 2024-03-23T14:35:14.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.892+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d0146340 con 0x7f46d00b3c10 2024-03-23T14:35:14.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.892+0000 7f46ccff9640 1 -- 172.21.15.161:0/341338896 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f46c0007450 con 0x7f46d00b3c10 2024-03-23T14:35:14.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.892+0000 7f46ccff9640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46bc06b810 0x7f46bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:14.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.893+0000 7f46ccff9640 1 -- 172.21.15.161:0/341338896 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f46c0089690 con 0x7f46d00b3c10 2024-03-23T14:35:14.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.893+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46bc06b810 0x7f46bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:14.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.893+0000 7f46cf7fe640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46bc06b810 0x7f46bc06dcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f46c4036000 tx=0x7f46c403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:14.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:14.897+0000 7f46ccff9640 1 -- 172.21.15.161:0/341338896 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f46d0146340 con 0x7f46d00b3c10 2024-03-23T14:35:15.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.107+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f46d0146340 con 0x7f46d00b3c10 2024-03-23T14:35:15.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.772+0000 7f46ccff9640 1 -- 172.21.15.161:0/341338896 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v6) ==== 122+0+0 (secure 0 0 0) 0x7f46d0146340 con 0x7f46d00b3c10 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.774+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46bc06b810 msgr2=0x7f46bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.774+0000 7f46d64da640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46bc06b810 0x7f46bc06dcd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f46c4036000 tx=0x7f46c403b040 comp rx=0 tx=0).stop 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.774+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00b3c10 msgr2=0x7f46d006ce40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.774+0000 7f46d64da640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00b3c10 0x7f46d006ce40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f46c000c500 tx=0x7f46c000c9c0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.775+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 shutdown_connections 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.775+0000 7f46d64da640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f46bc06b810 0x7f46bc06dcd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.775+0000 7f46d64da640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46d015e1d0 0x7f46d01f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.775+0000 7f46d64da640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46d00cfee0 0x7f46d0147c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.775+0000 7f46d64da640 1 --2- 172.21.15.161:0/341338896 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46d00b3c10 0x7f46d006ce40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.775+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 >> 172.21.15.161:0/341338896 conn(0x7f46d006b4a0 msgr2=0x7f46d0160070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.775+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 shutdown_connections 2024-03-23T14:35:15.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.775+0000 7f46d64da640 1 -- 172.21.15.161:0/341338896 wait complete. 2024-03-23T14:35:15.782 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T14:35:15.782 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:15.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.887+0000 7f2e23a07640 1 Processor -- start 2024-03-23T14:35:15.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e23a07640 1 -- start start 2024-03-23T14:35:15.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e23a07640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c15e400 0x7f2e1c0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:15.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e23a07640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2e1c0cf9d0 0x7f2e1c0d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:15.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e23a07640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e1c0d0460 con 0x7f2e1c0d1470 2024-03-23T14:35:15.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e23a07640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2e1c0d1050 con 0x7f2e1c0cf9d0 2024-03-23T14:35:15.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e23a07640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1c0d06b0 con 0x7f2e1c15e400 2024-03-23T14:35:15.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e20fa2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2e1c0cf9d0 0x7f2e1c0d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e20fa2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2e1c0cf9d0 0x7f2e1c0d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40980/0 (socket says 172.21.15.161:40980) 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e21fa4640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2e1c0d1470 0x7f2e1c0b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49788/0 (socket says 172.21.15.161:49788) 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.888+0000 7f2e20fa2640 1 -- 172.21.15.161:0/4146458405 learned_addr learned my addr 172.21.15.161:0/4146458405 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.889+0000 7f2e217a3640 1 --2- 172.21.15.161:0/4146458405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c15e400 0x7f2e1c0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.889+0000 7f2e20fa2640 1 -- 172.21.15.161:0/4146458405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c15e400 msgr2=0x7f2e1c0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.889+0000 7f2e20fa2640 1 --2- 172.21.15.161:0/4146458405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c15e400 0x7f2e1c0cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.889+0000 7f2e20fa2640 1 -- 172.21.15.161:0/4146458405 >> v1:172.21.15.161:6789/0 conn(0x7f2e1c0d1470 legacy=0x7f2e1c0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.889+0000 7f2e13fff640 1 -- 172.21.15.161:0/4146458405 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2754311509 0 0) 0x7f2e1c0d0460 con 0x7f2e1c0d1470 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.889+0000 7f2e20fa2640 1 -- 172.21.15.161:0/4146458405 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c0d0990 con 0x7f2e1c0cf9d0 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.889+0000 7f2e20fa2640 1 --2- 172.21.15.161:0/4146458405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c0cf9d0 0x7f2e1c0d2430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009870 tx=0x7f2e0c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3c1b92cb7c3e0c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e13fff640 1 -- 172.21.15.161:0/4146458405 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c030840 con 0x7f2e1c0cf9d0 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e13fff640 1 -- 172.21.15.161:0/4146458405 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e0c0309a0 con 0x7f2e1c0cf9d0 2024-03-23T14:35:15.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e13fff640 1 -- 172.21.15.161:0/4146458405 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e0c030d30 con 0x7f2e1c0cf9d0 2024-03-23T14:35:15.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e23a07640 1 -- 172.21.15.161:0/4146458405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c0cf9d0 msgr2=0x7f2e1c0d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:15.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e23a07640 1 --2- 172.21.15.161:0/4146458405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c0cf9d0 0x7f2e1c0d2430 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c009870 tx=0x7f2e0c02b980 comp rx=0 tx=0).stop 2024-03-23T14:35:15.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e23a07640 1 -- 172.21.15.161:0/4146458405 shutdown_connections 2024-03-23T14:35:15.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e23a07640 1 --2- 172.21.15.161:0/4146458405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c0cf9d0 0x7f2e1c0d2430 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e23a07640 1 --2- 172.21.15.161:0/4146458405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c15e400 0x7f2e1c0cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e23a07640 1 -- 172.21.15.161:0/4146458405 >> 172.21.15.161:0/4146458405 conn(0x7f2e1c09d190 msgr2=0x7f2e1c0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:15.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e23a07640 1 -- 172.21.15.161:0/4146458405 shutdown_connections 2024-03-23T14:35:15.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.890+0000 7f2e23a07640 1 -- 172.21.15.161:0/4146458405 wait complete. 2024-03-23T14:35:15.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.891+0000 7f2e23a07640 1 Processor -- start 2024-03-23T14:35:15.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.891+0000 7f2e23a07640 1 -- start start 2024-03-23T14:35:15.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e23a07640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c0cf9d0 0x7f2e1c1f3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:15.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e23a07640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1c0d1470 0x7f2e1c0cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:15.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e23a07640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c15e400 0x7f2e1c1f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:15.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e23a07640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c0cd6e0 con 0x7f2e1c15e400 2024-03-23T14:35:15.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e23a07640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c1f42c0 con 0x7f2e1c0d1470 2024-03-23T14:35:15.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e23a07640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1c1f45a0 con 0x7f2e1c0cf9d0 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e21fa4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c15e400 0x7f2e1c1f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e21fa4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c15e400 0x7f2e1c1f4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:40992/0 (socket says 172.21.15.161:40992) 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e21fa4640 1 -- 172.21.15.161:0/1041123596 learned_addr learned my addr 172.21.15.161:0/1041123596 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e217a3640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c0cf9d0 0x7f2e1c1f3c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e20fa2640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1c0d1470 0x7f2e1c0cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e21fa4640 1 -- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c0cf9d0 msgr2=0x7f2e1c1f3c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e21fa4640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c0cf9d0 0x7f2e1c1f3c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e21fa4640 1 -- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1c0d1470 msgr2=0x7f2e1c0cd1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e21fa4640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1c0d1470 0x7f2e1c0cd1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:15.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.892+0000 7f2e21fa4640 1 -- 172.21.15.161:0/1041123596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e0c0094d0 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.893+0000 7f2e21fa4640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c15e400 0x7f2e1c1f4e50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c450 tx=0x7f2e1400c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.893+0000 7f2e11ffb640 1 -- 172.21.15.161:0/1041123596 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e140074e0 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.893+0000 7f2e11ffb640 1 -- 172.21.15.161:0/1041123596 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e14007660 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.893+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1c1f4880 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.893+0000 7f2e11ffb640 1 -- 172.21.15.161:0/1041123596 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14004a20 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.893+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e1c0c9e50 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.894+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e1c0c8cb0 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.894+0000 7f2e11ffb640 1 -- 172.21.15.161:0/1041123596 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2e14004b80 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.895+0000 7f2e11ffb640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e0006b810 0x7f2e0006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.895+0000 7f2e11ffb640 1 -- 172.21.15.161:0/1041123596 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2e140895d0 con 0x7f2e1c15e400 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.895+0000 7f2e217a3640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e0006b810 0x7f2e0006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:15.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.895+0000 7f2e217a3640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e0006b810 0x7f2e0006dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e18011340 tx=0x7f2e1800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:15.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:15.899+0000 7f2e11ffb640 1 -- 172.21.15.161:0/1041123596 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2e1c0c8cb0 con 0x7f2e1c15e400 2024-03-23T14:35:16.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.107+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2e1c0c8cb0 con 0x7f2e1c15e400 2024-03-23T14:35:16.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.108+0000 7f2e11ffb640 1 -- 172.21.15.161:0/1041123596 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5181 (secure 0 0 0) 0x7f2e1c0c8cb0 con 0x7f2e1c15e400 2024-03-23T14:35:16.104 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:16.104 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:35:13.112471+0000","modified":"2024-03-23T14:35:15.108804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-23T14:35:16.104 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 6 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.109+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e0006b810 msgr2=0x7f2e0006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.109+0000 7f2e23a07640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e0006b810 0x7f2e0006dcd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e18011340 tx=0x7f2e1800b040 comp rx=0 tx=0).stop 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.109+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c15e400 msgr2=0x7f2e1c1f4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.109+0000 7f2e23a07640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c15e400 0x7f2e1c1f4e50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400c450 tx=0x7f2e1400c910 comp rx=0 tx=0).stop 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.110+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 shutdown_connections 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.110+0000 7f2e23a07640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e0006b810 0x7f2e0006dcd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.110+0000 7f2e23a07640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e1c15e400 0x7f2e1c1f4e50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.110+0000 7f2e23a07640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e1c0d1470 0x7f2e1c0cd1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.110+0000 7f2e23a07640 1 --2- 172.21.15.161:0/1041123596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e1c0cf9d0 0x7f2e1c1f3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.110+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 >> 172.21.15.161:0/1041123596 conn(0x7f2e1c09d190 msgr2=0x7f2e1c0b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:16.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.110+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 shutdown_connections 2024-03-23T14:35:16.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.110+0000 7f2e23a07640 1 -- 172.21.15.161:0/1041123596 wait complete. 2024-03-23T14:35:16.117 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:16.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.222+0000 7f24d441c640 1 Processor -- start 2024-03-23T14:35:16.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.222+0000 7f24d441c640 1 -- start start 2024-03-23T14:35:16.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.223+0000 7f24d441c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f24cc0d2b40 0x7f24cc0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.223+0000 7f24d441c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24cc0d0460 con 0x7f24cc0d1470 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.223+0000 7f24d441c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24cc0d06b0 con 0x7f24cc15e400 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.223+0000 7f24d441c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f24cc0d1050 con 0x7f24cc0d2b40 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d19b7640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f24cc0d1470 0x7f24cc0d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49800/0 (socket says 172.21.15.161:49800) 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d19b7640 1 -- 172.21.15.161:0/1732126012 learned_addr learned my addr 172.21.15.161:0/1732126012 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d29b9640 1 --2- 172.21.15.161:0/1732126012 >> v2:172.21.15.183:3300/0 conn(0x7f24cc0d2b40 0x7f24cc0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d11b6640 1 -- 172.21.15.161:0/1732126012 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1512475952 0 0) 0x7f24cc0d0460 con 0x7f24cc0d1470 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d11b6640 1 -- 172.21.15.161:0/1732126012 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24b8003520 con 0x7f24cc0d1470 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d11b6640 1 -- 172.21.15.161:0/1732126012 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1620266151 0 0) 0x7f24cc0d06b0 con 0x7f24cc15e400 2024-03-23T14:35:16.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d11b6640 1 -- 172.21.15.161:0/1732126012 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f24cc0d0460 con 0x7f24cc15e400 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d29b9640 1 -- 172.21.15.161:0/1732126012 >> v1:172.21.15.183:6789/0 conn(0x7f24cc15e400 legacy=0x7f24cc0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d29b9640 1 -- 172.21.15.161:0/1732126012 >> v1:172.21.15.161:6789/0 conn(0x7f24cc0d1470 legacy=0x7f24cc0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.224+0000 7f24d29b9640 1 -- 172.21.15.161:0/1732126012 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24cc0d0990 con 0x7f24cc0d2b40 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d29b9640 1 --2- 172.21.15.161:0/1732126012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 0x7f24cc0b0a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009d90 tx=0x7f24c402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c51d40c6729da78f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d11b6640 1 -- 172.21.15.161:0/1732126012 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c402cd00 con 0x7f24cc0d2b40 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d11b6640 1 -- 172.21.15.161:0/1732126012 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24c402ce60 con 0x7f24cc0d2b40 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d11b6640 1 -- 172.21.15.161:0/1732126012 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24c4034660 con 0x7f24cc0d2b40 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d441c640 1 -- 172.21.15.161:0/1732126012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 msgr2=0x7f24cc0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d441c640 1 --2- 172.21.15.161:0/1732126012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 0x7f24cc0b0a80 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f24c4009d90 tx=0x7f24c402bf50 comp rx=0 tx=0).stop 2024-03-23T14:35:16.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d441c640 1 -- 172.21.15.161:0/1732126012 shutdown_connections 2024-03-23T14:35:16.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d441c640 1 --2- 172.21.15.161:0/1732126012 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 0x7f24cc0b0a80 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d441c640 1 -- 172.21.15.161:0/1732126012 >> 172.21.15.161:0/1732126012 conn(0x7f24cc09d190 msgr2=0x7f24cc0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:16.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.225+0000 7f24d441c640 1 -- 172.21.15.161:0/1732126012 shutdown_connections 2024-03-23T14:35:16.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.226+0000 7f24d441c640 1 -- 172.21.15.161:0/1732126012 wait complete. 2024-03-23T14:35:16.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d441c640 1 Processor -- start 2024-03-23T14:35:16.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d441c640 1 -- start start 2024-03-23T14:35:16.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d441c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24cc0d1470 0x7f24cc15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d441c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 0x7f24cc0cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d441c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24cc15e400 0x7f24cc0cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d441c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc0cd020 con 0x7f24cc0d1470 2024-03-23T14:35:16.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d441c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f24cc0b6ca0 con 0x7f24cc0d2b40 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d441c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f24cc162a60 con 0x7f24cc15e400 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.227+0000 7f24d21b8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24cc0d1470 0x7f24cc15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d21b8640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24cc0d1470 0x7f24cc15b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41004/0 (socket says 172.21.15.161:41004) 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d21b8640 1 -- 172.21.15.161:0/1741703225 learned_addr learned my addr 172.21.15.161:0/1741703225 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d19b7640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 0x7f24cc0cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d29b9640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24cc15e400 0x7f24cc0cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d21b8640 1 -- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24cc15e400 msgr2=0x7f24cc0cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d21b8640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24cc15e400 0x7f24cc0cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d21b8640 1 -- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 msgr2=0x7f24cc0cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d21b8640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 0x7f24cc0cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d21b8640 1 -- 172.21.15.161:0/1741703225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c4009a70 con 0x7f24cc0d1470 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d21b8640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24cc0d1470 0x7f24cc15b210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00c500 tx=0x7f24bc00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24c2ffd640 1 -- 172.21.15.161:0/1741703225 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc0052d0 con 0x7f24cc0d1470 2024-03-23T14:35:16.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24c2ffd640 1 -- 172.21.15.161:0/1741703225 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f24bc005450 con 0x7f24cc0d1470 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24cc0cc4a0 con 0x7f24cc0d1470 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24c2ffd640 1 -- 172.21.15.161:0/1741703225 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24bc012870 con 0x7f24cc0d1470 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.228+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24cc0cca60 con 0x7f24cc0d1470 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.229+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24cc0a1e00 con 0x7f24cc0d1470 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.230+0000 7f24c2ffd640 1 -- 172.21.15.161:0/1741703225 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f24bc0129d0 con 0x7f24cc0d1470 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.230+0000 7f24c2ffd640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f24b006b810 0x7f24b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.230+0000 7f24c2ffd640 1 -- 172.21.15.161:0/1741703225 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f24bc089cc0 con 0x7f24cc0d1470 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.230+0000 7f24d19b7640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f24b006b810 0x7f24b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.231+0000 7f24d19b7640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f24b006b810 0x7f24b006dcd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0b4e90 tx=0x7f24c8007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.234+0000 7f24c2ffd640 1 -- 172.21.15.161:0/1741703225 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f24bc0547a0 con 0x7f24cc0d1470 2024-03-23T14:35:16.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.409+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f24cc04f550 con 0x7f24cc0d1470 2024-03-23T14:35:16.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.410+0000 7f24c2ffd640 1 -- 172.21.15.161:0/1741703225 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f24bc057e60 con 0x7f24cc0d1470 2024-03-23T14:35:16.406 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:16.406 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f24b006b810 msgr2=0x7f24b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f24b006b810 0x7f24b006dcd0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f24cc0b4e90 tx=0x7f24c8007d30 comp rx=0 tx=0).stop 2024-03-23T14:35:16.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24cc0d1470 msgr2=0x7f24cc15b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24cc0d1470 0x7f24cc15b210 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f24bc00c500 tx=0x7f24bc00c9c0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 shutdown_connections 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f24b006b810 0x7f24b006dcd0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f24cc15e400 0x7f24cc0cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f24cc0d2b40 0x7f24cc0cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 --2- 172.21.15.161:0/1741703225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f24cc0d1470 0x7f24cc15b210 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 >> 172.21.15.161:0/1741703225 conn(0x7f24cc09d190 msgr2=0x7f24cc15c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 shutdown_connections 2024-03-23T14:35:16.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.412+0000 7f24d441c640 1 -- 172.21.15.161:0/1741703225 wait complete. 2024-03-23T14:35:16.419 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T14:35:16.419 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:35:16.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.524+0000 7faf84455640 1 Processor -- start 2024-03-23T14:35:16.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.524+0000 7faf84455640 1 -- start start 2024-03-23T14:35:16.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf84455640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c145080 0x7faf7c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf84455640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faf7c1601f0 0x7faf7c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf84455640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf7c0cf4f0 con 0x7faf7c1607d0 2024-03-23T14:35:16.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf84455640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7faf7c071010 con 0x7faf7c1601f0 2024-03-23T14:35:16.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf84455640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7faf7c0cef30 con 0x7faf7c145080 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf829f2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faf7c1601f0 0x7faf7c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf829f2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faf7c1601f0 0x7faf7c15c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41006/0 (socket says 172.21.15.161:41006) 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf829f2640 1 -- 172.21.15.161:0/2058312149 learned_addr learned my addr 172.21.15.161:0/2058312149 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf819f0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7faf7c1607d0 0x7faf7c15fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49806/0 (socket says 172.21.15.161:49806) 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf821f1640 1 --2- 172.21.15.161:0/2058312149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c145080 0x7faf7c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf821f1640 1 -- 172.21.15.161:0/2058312149 >> v2:172.21.15.161:3300/0 conn(0x7faf7c1601f0 msgr2=0x7faf7c15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf821f1640 1 --2- 172.21.15.161:0/2058312149 >> v2:172.21.15.161:3300/0 conn(0x7faf7c1601f0 0x7faf7c15c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf821f1640 1 -- 172.21.15.161:0/2058312149 >> v1:172.21.15.161:6789/0 conn(0x7faf7c1607d0 legacy=0x7faf7c15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf821f1640 1 -- 172.21.15.161:0/2058312149 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf7c0cf210 con 0x7faf7c145080 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf811ef640 1 -- 172.21.15.161:0/2058312149 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3866799735 0 0) 0x7faf7c0cf4f0 con 0x7faf7c1607d0 2024-03-23T14:35:16.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.525+0000 7faf829f2640 1 --2- 172.21.15.161:0/2058312149 >> v2:172.21.15.161:3300/0 conn(0x7faf7c1601f0 0x7faf7c15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.526+0000 7faf821f1640 1 --2- 172.21.15.161:0/2058312149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c145080 0x7faf7c072a00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faf780097a0 tx=0x7faf7802b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb5a7070efb12108 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.526+0000 7faf811ef640 1 -- 172.21.15.161:0/2058312149 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78031780 con 0x7faf7c145080 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.526+0000 7faf811ef640 1 -- 172.21.15.161:0/2058312149 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf780318e0 con 0x7faf7c145080 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.526+0000 7faf811ef640 1 -- 172.21.15.161:0/2058312149 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78031c70 con 0x7faf7c145080 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.527+0000 7faf84455640 1 -- 172.21.15.161:0/2058312149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c145080 msgr2=0x7faf7c072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.527+0000 7faf84455640 1 --2- 172.21.15.161:0/2058312149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c145080 0x7faf7c072a00 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faf780097a0 tx=0x7faf7802b820 comp rx=0 tx=0).stop 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.527+0000 7faf84455640 1 -- 172.21.15.161:0/2058312149 shutdown_connections 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.527+0000 7faf84455640 1 --2- 172.21.15.161:0/2058312149 >> v2:172.21.15.161:3300/0 conn(0x7faf7c1601f0 0x7faf7c15c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.527+0000 7faf84455640 1 --2- 172.21.15.161:0/2058312149 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c145080 0x7faf7c072a00 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.527+0000 7faf84455640 1 -- 172.21.15.161:0/2058312149 >> 172.21.15.161:0/2058312149 conn(0x7faf7c0b17f0 msgr2=0x7faf7c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.527+0000 7faf84455640 1 -- 172.21.15.161:0/2058312149 shutdown_connections 2024-03-23T14:35:16.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.527+0000 7faf84455640 1 -- 172.21.15.161:0/2058312149 wait complete. 2024-03-23T14:35:16.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.528+0000 7faf84455640 1 Processor -- start 2024-03-23T14:35:16.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.528+0000 7faf84455640 1 -- start start 2024-03-23T14:35:16.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf84455640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf7c145080 0x7faf7c13ff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf84455640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7c1601f0 0x7faf7c1f1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf84455640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c1607d0 0x7faf7c1f4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf84455640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c13b8a0 con 0x7faf7c145080 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf84455640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faf7c13b4e0 con 0x7faf7c1601f0 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf84455640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7faf7c1f56a0 con 0x7faf7c1607d0 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf7c145080 0x7faf7c13ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf7c145080 0x7faf7c13ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41016/0 (socket says 172.21.15.161:41016) 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 -- 172.21.15.161:0/3801060809 learned_addr learned my addr 172.21.15.161:0/3801060809 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf819f0640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7c1601f0 0x7faf7c1f1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf829f2640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c1607d0 0x7faf7c1f4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 -- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c1607d0 msgr2=0x7faf7c1f4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c1607d0 0x7faf7c1f4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 -- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7c1601f0 msgr2=0x7faf7c1f1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7c1601f0 0x7faf7c1f1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 -- 172.21.15.161:0/3801060809 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf74009b30 con 0x7faf7c145080 2024-03-23T14:35:16.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf829f2640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c1607d0 0x7faf7c1f4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf821f1640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf7c145080 0x7faf7c13ff20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faf78034040 tx=0x7faf780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.529+0000 7faf72ffd640 1 -- 172.21.15.161:0/3801060809 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78032820 con 0x7faf7c145080 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.530+0000 7faf72ffd640 1 -- 172.21.15.161:0/3801060809 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf780329a0 con 0x7faf7c145080 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.530+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf78009400 con 0x7faf7c145080 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.530+0000 7faf72ffd640 1 -- 172.21.15.161:0/3801060809 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf78032d50 con 0x7faf7c145080 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.530+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf7c1f5c60 con 0x7faf7c145080 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.531+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf7c06cb70 con 0x7faf7c145080 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.531+0000 7faf72ffd640 1 -- 172.21.15.161:0/3801060809 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faf7804d3f0 con 0x7faf7c145080 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.531+0000 7faf72ffd640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faf6006b810 0x7faf6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.532+0000 7faf819f0640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faf6006b810 0x7faf6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.532+0000 7faf72ffd640 1 -- 172.21.15.161:0/3801060809 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7faf780abd70 con 0x7faf7c145080 2024-03-23T14:35:16.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.532+0000 7faf819f0640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faf6006b810 0x7faf6006dcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf7c15f5b0 tx=0x7faf6c007de0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.536+0000 7faf72ffd640 1 -- 172.21.15.161:0/3801060809 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faf78076790 con 0x7faf7c145080 2024-03-23T14:35:16.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.705+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7faf7c145460 con 0x7faf6006b810 2024-03-23T14:35:16.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.718+0000 7faf72ffd640 1 -- 172.21.15.161:0/3801060809 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251945 (secure 0 0 0) 0x7faf6c004c70 con 0x7faf6006b810 2024-03-23T14:35:16.717 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:16.718 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:35:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.720+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faf6006b810 msgr2=0x7faf6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.720+0000 7faf84455640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faf6006b810 0x7faf6006dcd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7faf7c15f5b0 tx=0x7faf6c007de0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.720+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf7c145080 msgr2=0x7faf7c13ff20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.720+0000 7faf84455640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf7c145080 0x7faf7c13ff20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7faf78034040 tx=0x7faf780046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.720+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 shutdown_connections 2024-03-23T14:35:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.721+0000 7faf84455640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faf6006b810 0x7faf6006dcd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.721+0000 7faf84455640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faf7c1607d0 0x7faf7c1f4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.721+0000 7faf84455640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faf7c1601f0 0x7faf7c1f1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.721+0000 7faf84455640 1 --2- 172.21.15.161:0/3801060809 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faf7c145080 0x7faf7c13ff20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.721+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 >> 172.21.15.161:0/3801060809 conn(0x7faf7c0b17f0 msgr2=0x7faf7c0af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:16.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.721+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 shutdown_connections 2024-03-23T14:35:16.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.721+0000 7faf84455640 1 -- 172.21.15.161:0/3801060809 wait complete. 2024-03-23T14:35:16.727 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-23T14:35:16.474840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219016,"kb_used_data":2800,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755704,"statfs":{"total":773094113280,"available":772869840896,"internally_reserved":0,"allocated":2867200,"data_stored":1740428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.728276"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991991+0000","last_change":"2024-03-23T14:35:13.778548+0000","last_active":"2024-03-23T14:35:14.991991+0000","last_peered":"2024-03-23T14:35:14.991991+0000","last_clean":"2024-03-23T14:35:14.991991+0000","last_became_active":"2024-03-23T14:35:13.778166+0000","last_became_peered":"2024-03-23T14:35:13.778166+0000","last_unstale":"2024-03-23T14:35:14.991991+0000","last_undegraded":"2024-03-23T14:35:14.991991+0000","last_fullsized":"2024-03-23T14:35:14.991991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:19.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774593+0000","last_change":"2024-03-23T14:35:13.788107+0000","last_active":"2024-03-23T14:35:14.774593+0000","last_peered":"2024-03-23T14:35:14.774593+0000","last_clean":"2024-03-23T14:35:14.774593+0000","last_became_active":"2024-03-23T14:35:13.787705+0000","last_became_peered":"2024-03-23T14:35:13.787705+0000","last_unstale":"2024-03-23T14:35:14.774593+0000","last_undegraded":"2024-03-23T14:35:14.774593+0000","last_fullsized":"2024-03-23T14:35:14.774593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991567+0000","last_change":"2024-03-23T14:35:13.783735+0000","last_active":"2024-03-23T14:35:14.991567+0000","last_peered":"2024-03-23T14:35:14.991567+0000","last_clean":"2024-03-23T14:35:14.991567+0000","last_became_active":"2024-03-23T14:35:13.783349+0000","last_became_peered":"2024-03-23T14:35:13.783349+0000","last_unstale":"2024-03-23T14:35:14.991567+0000","last_undegraded":"2024-03-23T14:35:14.991567+0000","last_fullsized":"2024-03-23T14:35:14.991567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991261+0000","last_change":"2024-03-23T14:35:13.787909+0000","last_active":"2024-03-23T14:35:14.991261+0000","last_peered":"2024-03-23T14:35:14.991261+0000","last_clean":"2024-03-23T14:35:14.991261+0000","last_became_active":"2024-03-23T14:35:13.786041+0000","last_became_peered":"2024-03-23T14:35:13.786041+0000","last_unstale":"2024-03-23T14:35:14.991261+0000","last_undegraded":"2024-03-23T14:35:14.991261+0000","last_fullsized":"2024-03-23T14:35:14.991261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991979+0000","last_change":"2024-03-23T14:35:13.784230+0000","last_active":"2024-03-23T14:35:14.991979+0000","last_peered":"2024-03-23T14:35:14.991979+0000","last_clean":"2024-03-23T14:35:14.991979+0000","last_became_active":"2024-03-23T14:35:13.781446+0000","last_became_peered":"2024-03-23T14:35:13.781446+0000","last_unstale":"2024-03-23T14:35:14.991979+0000","last_undegraded":"2024-03-23T14:35:14.991979+0000","last_fullsized":"2024-03-23T14:35:14.991979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:20.569093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991109+0000","last_change":"2024-03-23T14:35:13.785787+0000","last_active":"2024-03-23T14:35:14.991109+0000","last_peered":"2024-03-23T14:35:14.991109+0000","last_clean":"2024-03-23T14:35:14.991109+0000","last_became_active":"2024-03-23T14:35:13.785528+0000","last_became_peered":"2024-03-23T14:35:13.785528+0000","last_unstale":"2024-03-23T14:35:14.991109+0000","last_undegraded":"2024-03-23T14:35:14.991109+0000","last_fullsized":"2024-03-23T14:35:14.991109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991672+0000","last_change":"2024-03-23T14:35:13.787315+0000","last_active":"2024-03-23T14:35:14.991672+0000","last_peered":"2024-03-23T14:35:14.991672+0000","last_clean":"2024-03-23T14:35:14.991672+0000","last_became_active":"2024-03-23T14:35:13.787032+0000","last_became_peered":"2024-03-23T14:35:13.787032+0000","last_unstale":"2024-03-23T14:35:14.991672+0000","last_undegraded":"2024-03-23T14:35:14.991672+0000","last_fullsized":"2024-03-23T14:35:14.991672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991225+0000","last_change":"2024-03-23T14:35:13.779324+0000","last_active":"2024-03-23T14:35:14.991225+0000","last_peered":"2024-03-23T14:35:14.991225+0000","last_clean":"2024-03-23T14:35:14.991225+0000","last_became_active":"2024-03-23T14:35:13.779048+0000","last_became_peered":"2024-03-23T14:35:13.779048+0000","last_unstale":"2024-03-23T14:35:14.991225+0000","last_undegraded":"2024-03-23T14:35:14.991225+0000","last_fullsized":"2024-03-23T14:35:14.991225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:37.554855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992038+0000","last_change":"2024-03-23T14:35:13.781031+0000","last_active":"2024-03-23T14:35:14.992038+0000","last_peered":"2024-03-23T14:35:14.992038+0000","last_clean":"2024-03-23T14:35:14.992038+0000","last_became_active":"2024-03-23T14:35:13.780762+0000","last_became_peered":"2024-03-23T14:35:13.780762+0000","last_unstale":"2024-03-23T14:35:14.992038+0000","last_undegraded":"2024-03-23T14:35:14.992038+0000","last_fullsized":"2024-03-23T14:35:14.992038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:17:27.919253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916812+0000","last_change":"2024-03-23T14:35:13.776904+0000","last_active":"2024-03-23T14:35:14.916812+0000","last_peered":"2024-03-23T14:35:14.916812+0000","last_clean":"2024-03-23T14:35:14.916812+0000","last_became_active":"2024-03-23T14:35:13.776499+0000","last_became_peered":"2024-03-23T14:35:13.776499+0000","last_unstale":"2024-03-23T14:35:14.916812+0000","last_undegraded":"2024-03-23T14:35:14.916812+0000","last_fullsized":"2024-03-23T14:35:14.916812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916738+0000","last_change":"2024-03-23T14:35:13.777465+0000","last_active":"2024-03-23T14:35:14.916738+0000","last_peered":"2024-03-23T14:35:14.916738+0000","last_clean":"2024-03-23T14:35:14.916738+0000","last_became_active":"2024-03-23T14:35:13.777169+0000","last_became_peered":"2024-03-23T14:35:13.777169+0000","last_unstale":"2024-03-23T14:35:14.916738+0000","last_undegraded":"2024-03-23T14:35:14.916738+0000","last_fullsized":"2024-03-23T14:35:14.916738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775225+0000","last_change":"2024-03-23T14:35:13.776334+0000","last_active":"2024-03-23T14:35:14.775225+0000","last_peered":"2024-03-23T14:35:14.775225+0000","last_clean":"2024-03-23T14:35:14.775225+0000","last_became_active":"2024-03-23T14:35:13.776069+0000","last_became_peered":"2024-03-23T14:35:13.776069+0000","last_unstale":"2024-03-23T14:35:14.775225+0000","last_undegraded":"2024-03-23T14:35:14.775225+0000","last_fullsized":"2024-03-23T14:35:14.775225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:44.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991775+0000","last_change":"2024-03-23T14:35:13.779105+0000","last_active":"2024-03-23T14:35:14.991775+0000","last_peered":"2024-03-23T14:35:14.991775+0000","last_clean":"2024-03-23T14:35:14.991775+0000","last_became_active":"2024-03-23T14:35:13.778836+0000","last_became_peered":"2024-03-23T14:35:13.778836+0000","last_unstale":"2024-03-23T14:35:14.991775+0000","last_undegraded":"2024-03-23T14:35:14.991775+0000","last_fullsized":"2024-03-23T14:35:14.991775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991202+0000","last_change":"2024-03-23T14:35:13.786094+0000","last_active":"2024-03-23T14:35:14.991202+0000","last_peered":"2024-03-23T14:35:14.991202+0000","last_clean":"2024-03-23T14:35:14.991202+0000","last_became_active":"2024-03-23T14:35:13.785749+0000","last_became_peered":"2024-03-23T14:35:13.785749+0000","last_unstale":"2024-03-23T14:35:14.991202+0000","last_undegraded":"2024-03-23T14:35:14.991202+0000","last_fullsized":"2024-03-23T14:35:14.991202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:48:45.763326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774304+0000","last_change":"2024-03-23T14:35:13.763672+0000","last_active":"2024-03-23T14:35:14.774304+0000","last_peered":"2024-03-23T14:35:14.774304+0000","last_clean":"2024-03-23T14:35:14.774304+0000","last_became_active":"2024-03-23T14:35:13.763446+0000","last_became_peered":"2024-03-23T14:35:13.763446+0000","last_unstale":"2024-03-23T14:35:14.774304+0000","last_undegraded":"2024-03-23T14:35:14.774304+0000","last_fullsized":"2024-03-23T14:35:14.774304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.990985+0000","last_change":"2024-03-23T14:35:13.788261+0000","last_active":"2024-03-23T14:35:14.990985+0000","last_peered":"2024-03-23T14:35:14.990985+0000","last_clean":"2024-03-23T14:35:14.990985+0000","last_became_active":"2024-03-23T14:35:13.787931+0000","last_became_peered":"2024-03-23T14:35:13.787931+0000","last_unstale":"2024-03-23T14:35:14.990985+0000","last_undegraded":"2024-03-23T14:35:14.990985+0000","last_fullsized":"2024-03-23T14:35:14.990985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991568+0000","last_change":"2024-03-23T14:35:13.788504+0000","last_active":"2024-03-23T14:35:14.991568+0000","last_peered":"2024-03-23T14:35:14.991568+0000","last_clean":"2024-03-23T14:35:14.991568+0000","last_became_active":"2024-03-23T14:35:13.788220+0000","last_became_peered":"2024-03-23T14:35:13.788220+0000","last_unstale":"2024-03-23T14:35:14.991568+0000","last_undegraded":"2024-03-23T14:35:14.991568+0000","last_fullsized":"2024-03-23T14:35:14.991568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:25.033547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771552+0000","last_change":"2024-03-23T14:35:13.777872+0000","last_active":"2024-03-23T14:35:14.771552+0000","last_peered":"2024-03-23T14:35:14.771552+0000","last_clean":"2024-03-23T14:35:14.771552+0000","last_became_active":"2024-03-23T14:35:13.777618+0000","last_became_peered":"2024-03-23T14:35:13.777618+0000","last_unstale":"2024-03-23T14:35:14.771552+0000","last_undegraded":"2024-03-23T14:35:14.771552+0000","last_fullsized":"2024-03-23T14:35:14.771552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:51.829515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992132+0000","last_change":"2024-03-23T14:35:13.785636+0000","last_active":"2024-03-23T14:35:14.992132+0000","last_peered":"2024-03-23T14:35:14.992132+0000","last_clean":"2024-03-23T14:35:14.992132+0000","last_became_active":"2024-03-23T14:35:13.784525+0000","last_became_peered":"2024-03-23T14:35:13.784525+0000","last_unstale":"2024-03-23T14:35:14.992132+0000","last_undegraded":"2024-03-23T14:35:14.992132+0000","last_fullsized":"2024-03-23T14:35:14.992132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:25.033547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916605+0000","last_change":"2024-03-23T14:35:13.775546+0000","last_active":"2024-03-23T14:35:14.916605+0000","last_peered":"2024-03-23T14:35:14.916605+0000","last_clean":"2024-03-23T14:35:14.916605+0000","last_became_active":"2024-03-23T14:35:13.774705+0000","last_became_peered":"2024-03-23T14:35:13.774705+0000","last_unstale":"2024-03-23T14:35:14.916605+0000","last_undegraded":"2024-03-23T14:35:14.916605+0000","last_fullsized":"2024-03-23T14:35:14.916605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991370+0000","last_change":"2024-03-23T14:35:13.785890+0000","last_active":"2024-03-23T14:35:14.991370+0000","last_peered":"2024-03-23T14:35:14.991370+0000","last_clean":"2024-03-23T14:35:14.991370+0000","last_became_active":"2024-03-23T14:35:13.785564+0000","last_became_peered":"2024-03-23T14:35:13.785564+0000","last_unstale":"2024-03-23T14:35:14.991370+0000","last_undegraded":"2024-03-23T14:35:14.991370+0000","last_fullsized":"2024-03-23T14:35:14.991370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:16:34.110338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991141+0000","last_change":"2024-03-23T14:35:13.785106+0000","last_active":"2024-03-23T14:35:14.991141+0000","last_peered":"2024-03-23T14:35:14.991141+0000","last_clean":"2024-03-23T14:35:14.991141+0000","last_became_active":"2024-03-23T14:35:13.784543+0000","last_became_peered":"2024-03-23T14:35:13.784543+0000","last_unstale":"2024-03-23T14:35:14.991141+0000","last_undegraded":"2024-03-23T14:35:14.991141+0000","last_fullsized":"2024-03-23T14:35:14.991141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991371+0000","last_change":"2024-03-23T14:35:13.782489+0000","last_active":"2024-03-23T14:35:14.991371+0000","last_peered":"2024-03-23T14:35:14.991371+0000","last_clean":"2024-03-23T14:35:14.991371+0000","last_became_active":"2024-03-23T14:35:13.782292+0000","last_became_peered":"2024-03-23T14:35:13.782292+0000","last_unstale":"2024-03-23T14:35:14.991371+0000","last_undegraded":"2024-03-23T14:35:14.991371+0000","last_fullsized":"2024-03-23T14:35:14.991371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:16:34.110338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991174+0000","last_change":"2024-03-23T14:35:13.776346+0000","last_active":"2024-03-23T14:35:14.991174+0000","last_peered":"2024-03-23T14:35:14.991174+0000","last_clean":"2024-03-23T14:35:14.991174+0000","last_became_active":"2024-03-23T14:35:13.776085+0000","last_became_peered":"2024-03-23T14:35:13.776085+0000","last_unstale":"2024-03-23T14:35:14.991174+0000","last_undegraded":"2024-03-23T14:35:14.991174+0000","last_fullsized":"2024-03-23T14:35:14.991174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992164+0000","last_change":"2024-03-23T14:35:13.781666+0000","last_active":"2024-03-23T14:35:14.992164+0000","last_peered":"2024-03-23T14:35:14.992164+0000","last_clean":"2024-03-23T14:35:14.992164+0000","last_became_active":"2024-03-23T14:35:13.781380+0000","last_became_peered":"2024-03-23T14:35:13.781380+0000","last_unstale":"2024-03-23T14:35:14.992164+0000","last_undegraded":"2024-03-23T14:35:14.992164+0000","last_fullsized":"2024-03-23T14:35:14.992164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916837+0000","last_change":"2024-03-23T14:35:13.774246+0000","last_active":"2024-03-23T14:35:14.916837+0000","last_peered":"2024-03-23T14:35:14.916837+0000","last_clean":"2024-03-23T14:35:14.916837+0000","last_became_active":"2024-03-23T14:35:13.773945+0000","last_became_peered":"2024-03-23T14:35:13.773945+0000","last_unstale":"2024-03-23T14:35:14.916837+0000","last_undegraded":"2024-03-23T14:35:14.916837+0000","last_fullsized":"2024-03-23T14:35:14.916837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774384+0000","last_change":"2024-03-23T14:35:13.774154+0000","last_active":"2024-03-23T14:35:14.774384+0000","last_peered":"2024-03-23T14:35:14.774384+0000","last_clean":"2024-03-23T14:35:14.774384+0000","last_became_active":"2024-03-23T14:35:13.773320+0000","last_became_peered":"2024-03-23T14:35:13.773320+0000","last_unstale":"2024-03-23T14:35:14.774384+0000","last_undegraded":"2024-03-23T14:35:14.774384+0000","last_fullsized":"2024-03-23T14:35:14.774384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991890+0000","last_change":"2024-03-23T14:35:13.782131+0000","last_active":"2024-03-23T14:35:14.991890+0000","last_peered":"2024-03-23T14:35:14.991890+0000","last_clean":"2024-03-23T14:35:14.991890+0000","last_became_active":"2024-03-23T14:35:13.781455+0000","last_became_peered":"2024-03-23T14:35:13.781455+0000","last_unstale":"2024-03-23T14:35:14.991890+0000","last_undegraded":"2024-03-23T14:35:14.991890+0000","last_fullsized":"2024-03-23T14:35:14.991890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991510+0000","last_change":"2024-03-23T14:35:13.787101+0000","last_active":"2024-03-23T14:35:14.991510+0000","last_peered":"2024-03-23T14:35:14.991510+0000","last_clean":"2024-03-23T14:35:14.991510+0000","last_became_active":"2024-03-23T14:35:13.786742+0000","last_became_peered":"2024-03-23T14:35:13.786742+0000","last_unstale":"2024-03-23T14:35:14.991510+0000","last_undegraded":"2024-03-23T14:35:14.991510+0000","last_fullsized":"2024-03-23T14:35:14.991510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991510+0000","last_change":"2024-03-23T14:35:13.781923+0000","last_active":"2024-03-23T14:35:14.991510+0000","last_peered":"2024-03-23T14:35:14.991510+0000","last_clean":"2024-03-23T14:35:14.991510+0000","last_became_active":"2024-03-23T14:35:13.781553+0000","last_became_peered":"2024-03-23T14:35:13.781553+0000","last_unstale":"2024-03-23T14:35:14.991510+0000","last_undegraded":"2024-03-23T14:35:14.991510+0000","last_fullsized":"2024-03-23T14:35:14.991510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:31:20.667310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771511+0000","last_change":"2024-03-23T14:35:13.775171+0000","last_active":"2024-03-23T14:35:14.771511+0000","last_peered":"2024-03-23T14:35:14.771511+0000","last_clean":"2024-03-23T14:35:14.771511+0000","last_became_active":"2024-03-23T14:35:13.774898+0000","last_became_peered":"2024-03-23T14:35:13.774898+0000","last_unstale":"2024-03-23T14:35:14.771511+0000","last_undegraded":"2024-03-23T14:35:14.771511+0000","last_fullsized":"2024-03-23T14:35:14.771511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:44.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916695+0000","last_change":"2024-03-23T14:35:13.785730+0000","last_active":"2024-03-23T14:35:14.916695+0000","last_peered":"2024-03-23T14:35:14.916695+0000","last_clean":"2024-03-23T14:35:14.916695+0000","last_became_active":"2024-03-23T14:35:13.785427+0000","last_became_peered":"2024-03-23T14:35:13.785427+0000","last_unstale":"2024-03-23T14:35:14.916695+0000","last_undegraded":"2024-03-23T14:35:14.916695+0000","last_fullsized":"2024-03-23T14:35:14.916695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774946+0000","last_change":"2024-03-23T14:35:13.769227+0000","last_active":"2024-03-23T14:35:14.774946+0000","last_peered":"2024-03-23T14:35:14.774946+0000","last_clean":"2024-03-23T14:35:14.774946+0000","last_became_active":"2024-03-23T14:35:13.768866+0000","last_became_peered":"2024-03-23T14:35:13.768866+0000","last_unstale":"2024-03-23T14:35:14.774946+0000","last_undegraded":"2024-03-23T14:35:14.774946+0000","last_fullsized":"2024-03-23T14:35:14.774946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:26.689847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992031+0000","last_change":"2024-03-23T14:35:12.777511+0000","last_active":"2024-03-23T14:35:14.992031+0000","last_peered":"2024-03-23T14:35:14.992031+0000","last_clean":"2024-03-23T14:35:14.992031+0000","last_became_active":"2024-03-23T14:35:12.777173+0000","last_became_peered":"2024-03-23T14:35:12.777173+0000","last_unstale":"2024-03-23T14:35:14.992031+0000","last_undegraded":"2024-03-23T14:35:14.992031+0000","last_fullsized":"2024-03-23T14:35:14.992031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:50.295934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991649+0000","last_change":"2024-03-23T14:35:13.784656+0000","last_active":"2024-03-23T14:35:14.991649+0000","last_peered":"2024-03-23T14:35:14.991649+0000","last_clean":"2024-03-23T14:35:14.991649+0000","last_became_active":"2024-03-23T14:35:13.783372+0000","last_became_peered":"2024-03-23T14:35:13.783372+0000","last_unstale":"2024-03-23T14:35:14.991649+0000","last_undegraded":"2024-03-23T14:35:14.991649+0000","last_fullsized":"2024-03-23T14:35:14.991649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916556+0000","last_change":"2024-03-23T14:35:12.772860+0000","last_active":"2024-03-23T14:35:14.916556+0000","last_peered":"2024-03-23T14:35:14.916556+0000","last_clean":"2024-03-23T14:35:14.916556+0000","last_became_active":"2024-03-23T14:35:12.772437+0000","last_became_peered":"2024-03-23T14:35:12.772437+0000","last_unstale":"2024-03-23T14:35:14.916556+0000","last_undegraded":"2024-03-23T14:35:14.916556+0000","last_fullsized":"2024-03-23T14:35:14.916556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991525+0000","last_change":"2024-03-23T14:35:12.773627+0000","last_active":"2024-03-23T14:35:14.991525+0000","last_peered":"2024-03-23T14:35:14.991525+0000","last_clean":"2024-03-23T14:35:14.991525+0000","last_became_active":"2024-03-23T14:35:12.773323+0000","last_became_peered":"2024-03-23T14:35:12.773323+0000","last_unstale":"2024-03-23T14:35:14.991525+0000","last_undegraded":"2024-03-23T14:35:14.991525+0000","last_fullsized":"2024-03-23T14:35:14.991525+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991373+0000","last_change":"2024-03-23T14:35:13.786951+0000","last_active":"2024-03-23T14:35:14.991373+0000","last_peered":"2024-03-23T14:35:14.991373+0000","last_clean":"2024-03-23T14:35:14.991373+0000","last_became_active":"2024-03-23T14:35:13.786684+0000","last_became_peered":"2024-03-23T14:35:13.786684+0000","last_unstale":"2024-03-23T14:35:14.991373+0000","last_undegraded":"2024-03-23T14:35:14.991373+0000","last_fullsized":"2024-03-23T14:35:14.991373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991479+0000","last_change":"2024-03-23T14:35:12.775888+0000","last_active":"2024-03-23T14:35:14.991479+0000","last_peered":"2024-03-23T14:35:14.991479+0000","last_clean":"2024-03-23T14:35:14.991479+0000","last_became_active":"2024-03-23T14:35:12.775610+0000","last_became_peered":"2024-03-23T14:35:12.775610+0000","last_unstale":"2024-03-23T14:35:14.991479+0000","last_undegraded":"2024-03-23T14:35:14.991479+0000","last_fullsized":"2024-03-23T14:35:14.991479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774293+0000","last_change":"2024-03-23T14:35:13.778143+0000","last_active":"2024-03-23T14:35:14.774293+0000","last_peered":"2024-03-23T14:35:14.774293+0000","last_clean":"2024-03-23T14:35:14.774293+0000","last_became_active":"2024-03-23T14:35:13.776770+0000","last_became_peered":"2024-03-23T14:35:13.776770+0000","last_unstale":"2024-03-23T14:35:14.774293+0000","last_undegraded":"2024-03-23T14:35:14.774293+0000","last_fullsized":"2024-03-23T14:35:14.774293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916781+0000","last_change":"2024-03-23T14:35:12.776956+0000","last_active":"2024-03-23T14:35:14.916781+0000","last_peered":"2024-03-23T14:35:14.916781+0000","last_clean":"2024-03-23T14:35:14.916781+0000","last_became_active":"2024-03-23T14:35:12.776658+0000","last_became_peered":"2024-03-23T14:35:12.776658+0000","last_unstale":"2024-03-23T14:35:14.916781+0000","last_undegraded":"2024-03-23T14:35:14.916781+0000","last_fullsized":"2024-03-23T14:35:14.916781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991420+0000","last_change":"2024-03-23T14:35:13.786795+0000","last_active":"2024-03-23T14:35:14.991420+0000","last_peered":"2024-03-23T14:35:14.991420+0000","last_clean":"2024-03-23T14:35:14.991420+0000","last_became_active":"2024-03-23T14:35:13.786533+0000","last_became_peered":"2024-03-23T14:35:13.786533+0000","last_unstale":"2024-03-23T14:35:14.991420+0000","last_undegraded":"2024-03-23T14:35:14.991420+0000","last_fullsized":"2024-03-23T14:35:14.991420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:19.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991421+0000","last_change":"2024-03-23T14:35:12.779292+0000","last_active":"2024-03-23T14:35:14.991421+0000","last_peered":"2024-03-23T14:35:14.991421+0000","last_clean":"2024-03-23T14:35:14.991421+0000","last_became_active":"2024-03-23T14:35:12.778957+0000","last_became_peered":"2024-03-23T14:35:12.778957+0000","last_unstale":"2024-03-23T14:35:14.991421+0000","last_undegraded":"2024-03-23T14:35:14.991421+0000","last_fullsized":"2024-03-23T14:35:14.991421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991873+0000","last_change":"2024-03-23T14:35:13.785374+0000","last_active":"2024-03-23T14:35:14.991873+0000","last_peered":"2024-03-23T14:35:14.991873+0000","last_clean":"2024-03-23T14:35:14.991873+0000","last_became_active":"2024-03-23T14:35:13.784972+0000","last_became_peered":"2024-03-23T14:35:13.784972+0000","last_unstale":"2024-03-23T14:35:14.991873+0000","last_undegraded":"2024-03-23T14:35:14.991873+0000","last_fullsized":"2024-03-23T14:35:14.991873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774494+0000","last_change":"2024-03-23T14:35:12.775269+0000","last_active":"2024-03-23T14:35:14.774494+0000","last_peered":"2024-03-23T14:35:14.774494+0000","last_clean":"2024-03-23T14:35:14.774494+0000","last_became_active":"2024-03-23T14:35:12.774985+0000","last_became_peered":"2024-03-23T14:35:12.774985+0000","last_unstale":"2024-03-23T14:35:14.774494+0000","last_undegraded":"2024-03-23T14:35:14.774494+0000","last_fullsized":"2024-03-23T14:35:14.774494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775426+0000","last_change":"2024-03-23T14:35:13.777078+0000","last_active":"2024-03-23T14:35:14.775426+0000","last_peered":"2024-03-23T14:35:14.775426+0000","last_clean":"2024-03-23T14:35:14.775426+0000","last_became_active":"2024-03-23T14:35:13.776801+0000","last_became_peered":"2024-03-23T14:35:13.776801+0000","last_unstale":"2024-03-23T14:35:14.775426+0000","last_undegraded":"2024-03-23T14:35:14.775426+0000","last_fullsized":"2024-03-23T14:35:14.775426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:51.829515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916705+0000","last_change":"2024-03-23T14:35:12.773980+0000","last_active":"2024-03-23T14:35:14.916705+0000","last_peered":"2024-03-23T14:35:14.916705+0000","last_clean":"2024-03-23T14:35:14.916705+0000","last_became_active":"2024-03-23T14:35:12.773168+0000","last_became_peered":"2024-03-23T14:35:12.773168+0000","last_unstale":"2024-03-23T14:35:14.916705+0000","last_undegraded":"2024-03-23T14:35:14.916705+0000","last_fullsized":"2024-03-23T14:35:14.916705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991870+0000","last_change":"2024-03-23T14:35:13.789833+0000","last_active":"2024-03-23T14:35:14.991870+0000","last_peered":"2024-03-23T14:35:14.991870+0000","last_clean":"2024-03-23T14:35:14.991870+0000","last_became_active":"2024-03-23T14:35:13.789586+0000","last_became_peered":"2024-03-23T14:35:13.789586+0000","last_unstale":"2024-03-23T14:35:14.991870+0000","last_undegraded":"2024-03-23T14:35:14.991870+0000","last_fullsized":"2024-03-23T14:35:14.991870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991752+0000","last_change":"2024-03-23T14:35:12.773078+0000","last_active":"2024-03-23T14:35:14.991752+0000","last_peered":"2024-03-23T14:35:14.991752+0000","last_clean":"2024-03-23T14:35:14.991752+0000","last_became_active":"2024-03-23T14:35:12.772785+0000","last_became_peered":"2024-03-23T14:35:12.772785+0000","last_unstale":"2024-03-23T14:35:14.991752+0000","last_undegraded":"2024-03-23T14:35:14.991752+0000","last_fullsized":"2024-03-23T14:35:14.991752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:33:10.353085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991346+0000","last_change":"2024-03-23T14:35:13.784377+0000","last_active":"2024-03-23T14:35:14.991346+0000","last_peered":"2024-03-23T14:35:14.991346+0000","last_clean":"2024-03-23T14:35:14.991346+0000","last_became_active":"2024-03-23T14:35:13.783490+0000","last_became_peered":"2024-03-23T14:35:13.783490+0000","last_unstale":"2024-03-23T14:35:14.991346+0000","last_undegraded":"2024-03-23T14:35:14.991346+0000","last_fullsized":"2024-03-23T14:35:14.991346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:37.554855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991229+0000","last_change":"2024-03-23T14:35:12.780946+0000","last_active":"2024-03-23T14:35:14.991229+0000","last_peered":"2024-03-23T14:35:14.991229+0000","last_clean":"2024-03-23T14:35:14.991229+0000","last_became_active":"2024-03-23T14:35:12.780648+0000","last_became_peered":"2024-03-23T14:35:12.780648+0000","last_unstale":"2024-03-23T14:35:14.991229+0000","last_undegraded":"2024-03-23T14:35:14.991229+0000","last_fullsized":"2024-03-23T14:35:14.991229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992120+0000","last_change":"2024-03-23T14:35:13.784062+0000","last_active":"2024-03-23T14:35:14.992120+0000","last_peered":"2024-03-23T14:35:14.992120+0000","last_clean":"2024-03-23T14:35:14.992120+0000","last_became_active":"2024-03-23T14:35:13.783777+0000","last_became_peered":"2024-03-23T14:35:13.783777+0000","last_unstale":"2024-03-23T14:35:14.992120+0000","last_undegraded":"2024-03-23T14:35:14.992120+0000","last_fullsized":"2024-03-23T14:35:14.992120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:31:20.667310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916665+0000","last_change":"2024-03-23T14:35:12.772698+0000","last_active":"2024-03-23T14:35:14.916665+0000","last_peered":"2024-03-23T14:35:14.916665+0000","last_clean":"2024-03-23T14:35:14.916665+0000","last_became_active":"2024-03-23T14:35:12.772263+0000","last_became_peered":"2024-03-23T14:35:12.772263+0000","last_unstale":"2024-03-23T14:35:14.916665+0000","last_undegraded":"2024-03-23T14:35:14.916665+0000","last_fullsized":"2024-03-23T14:35:14.916665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916589+0000","last_change":"2024-03-23T14:35:13.785748+0000","last_active":"2024-03-23T14:35:14.916589+0000","last_peered":"2024-03-23T14:35:14.916589+0000","last_clean":"2024-03-23T14:35:14.916589+0000","last_became_active":"2024-03-23T14:35:13.785468+0000","last_became_peered":"2024-03-23T14:35:13.785468+0000","last_unstale":"2024-03-23T14:35:14.916589+0000","last_undegraded":"2024-03-23T14:35:14.916589+0000","last_fullsized":"2024-03-23T14:35:14.916589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771407+0000","last_change":"2024-03-23T14:35:12.766835+0000","last_active":"2024-03-23T14:35:14.771407+0000","last_peered":"2024-03-23T14:35:14.771407+0000","last_clean":"2024-03-23T14:35:14.771407+0000","last_became_active":"2024-03-23T14:35:12.766558+0000","last_became_peered":"2024-03-23T14:35:12.766558+0000","last_unstale":"2024-03-23T14:35:14.771407+0000","last_undegraded":"2024-03-23T14:35:14.771407+0000","last_fullsized":"2024-03-23T14:35:14.771407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:50.295934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991770+0000","last_change":"2024-03-23T14:35:13.783257+0000","last_active":"2024-03-23T14:35:14.991770+0000","last_peered":"2024-03-23T14:35:14.991770+0000","last_clean":"2024-03-23T14:35:14.991770+0000","last_became_active":"2024-03-23T14:35:13.782963+0000","last_became_peered":"2024-03-23T14:35:13.782963+0000","last_unstale":"2024-03-23T14:35:14.991770+0000","last_undegraded":"2024-03-23T14:35:14.991770+0000","last_fullsized":"2024-03-23T14:35:14.991770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775329+0000","last_change":"2024-03-23T14:35:12.769564+0000","last_active":"2024-03-23T14:35:14.775329+0000","last_peered":"2024-03-23T14:35:14.775329+0000","last_clean":"2024-03-23T14:35:14.775329+0000","last_became_active":"2024-03-23T14:35:12.769153+0000","last_became_peered":"2024-03-23T14:35:12.769153+0000","last_unstale":"2024-03-23T14:35:14.775329+0000","last_undegraded":"2024-03-23T14:35:14.775329+0000","last_fullsized":"2024-03-23T14:35:14.775329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:37:12.470551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991883+0000","last_change":"2024-03-23T14:35:13.783901+0000","last_active":"2024-03-23T14:35:14.991883+0000","last_peered":"2024-03-23T14:35:14.991883+0000","last_clean":"2024-03-23T14:35:14.991883+0000","last_became_active":"2024-03-23T14:35:13.783590+0000","last_became_peered":"2024-03-23T14:35:13.783590+0000","last_unstale":"2024-03-23T14:35:14.991883+0000","last_undegraded":"2024-03-23T14:35:14.991883+0000","last_fullsized":"2024-03-23T14:35:14.991883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771642+0000","last_change":"2024-03-23T14:35:12.767718+0000","last_active":"2024-03-23T14:35:14.771642+0000","last_peered":"2024-03-23T14:35:14.771642+0000","last_clean":"2024-03-23T14:35:14.771642+0000","last_became_active":"2024-03-23T14:35:12.767437+0000","last_became_peered":"2024-03-23T14:35:12.767437+0000","last_unstale":"2024-03-23T14:35:14.771642+0000","last_undegraded":"2024-03-23T14:35:14.771642+0000","last_fullsized":"2024-03-23T14:35:14.771642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992143+0000","last_change":"2024-03-23T14:35:13.785178+0000","last_active":"2024-03-23T14:35:14.992143+0000","last_peered":"2024-03-23T14:35:14.992143+0000","last_clean":"2024-03-23T14:35:14.992143+0000","last_became_active":"2024-03-23T14:35:13.781728+0000","last_became_peered":"2024-03-23T14:35:13.781728+0000","last_unstale":"2024-03-23T14:35:14.992143+0000","last_undegraded":"2024-03-23T14:35:14.992143+0000","last_fullsized":"2024-03-23T14:35:14.992143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:26.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774530+0000","last_change":"2024-03-23T14:35:12.764015+0000","last_active":"2024-03-23T14:35:14.774530+0000","last_peered":"2024-03-23T14:35:14.774530+0000","last_clean":"2024-03-23T14:35:14.774530+0000","last_became_active":"2024-03-23T14:35:12.763735+0000","last_became_peered":"2024-03-23T14:35:12.763735+0000","last_unstale":"2024-03-23T14:35:14.774530+0000","last_undegraded":"2024-03-23T14:35:14.774530+0000","last_fullsized":"2024-03-23T14:35:14.774530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:43.686652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991578+0000","last_change":"2024-03-23T14:35:13.780335+0000","last_active":"2024-03-23T14:35:14.991578+0000","last_peered":"2024-03-23T14:35:14.991578+0000","last_clean":"2024-03-23T14:35:14.991578+0000","last_became_active":"2024-03-23T14:35:13.780078+0000","last_became_peered":"2024-03-23T14:35:13.780078+0000","last_unstale":"2024-03-23T14:35:14.991578+0000","last_undegraded":"2024-03-23T14:35:14.991578+0000","last_fullsized":"2024-03-23T14:35:14.991578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:26.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771754+0000","last_change":"2024-03-23T14:35:12.772701+0000","last_active":"2024-03-23T14:35:14.771754+0000","last_peered":"2024-03-23T14:35:14.771754+0000","last_clean":"2024-03-23T14:35:14.771754+0000","last_became_active":"2024-03-23T14:35:12.772375+0000","last_became_peered":"2024-03-23T14:35:12.772375+0000","last_unstale":"2024-03-23T14:35:14.771754+0000","last_undegraded":"2024-03-23T14:35:14.771754+0000","last_fullsized":"2024-03-23T14:35:14.771754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991996+0000","last_change":"2024-03-23T14:35:13.789100+0000","last_active":"2024-03-23T14:35:14.991996+0000","last_peered":"2024-03-23T14:35:14.991996+0000","last_clean":"2024-03-23T14:35:14.991996+0000","last_became_active":"2024-03-23T14:35:13.788744+0000","last_became_peered":"2024-03-23T14:35:13.788744+0000","last_unstale":"2024-03-23T14:35:14.991996+0000","last_undegraded":"2024-03-23T14:35:14.991996+0000","last_fullsized":"2024-03-23T14:35:14.991996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:54:02.413325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991363+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:35:14.991363+0000","last_peered":"2024-03-23T14:35:14.991363+0000","last_clean":"2024-03-23T14:35:14.991363+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:35:14.991363+0000","last_undegraded":"2024-03-23T14:35:14.991363+0000","last_fullsized":"2024-03-23T14:35:14.991363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991027+0000","last_change":"2024-03-23T14:35:12.772428+0000","last_active":"2024-03-23T14:35:14.991027+0000","last_peered":"2024-03-23T14:35:14.991027+0000","last_clean":"2024-03-23T14:35:14.991027+0000","last_became_active":"2024-03-23T14:35:12.772152+0000","last_became_peered":"2024-03-23T14:35:12.772152+0000","last_unstale":"2024-03-23T14:35:14.991027+0000","last_undegraded":"2024-03-23T14:35:14.991027+0000","last_fullsized":"2024-03-23T14:35:14.991027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992036+0000","last_change":"2024-03-23T14:35:13.779880+0000","last_active":"2024-03-23T14:35:14.992036+0000","last_peered":"2024-03-23T14:35:14.992036+0000","last_clean":"2024-03-23T14:35:14.992036+0000","last_became_active":"2024-03-23T14:35:13.779608+0000","last_became_peered":"2024-03-23T14:35:13.779608+0000","last_unstale":"2024-03-23T14:35:14.992036+0000","last_undegraded":"2024-03-23T14:35:14.992036+0000","last_fullsized":"2024-03-23T14:35:14.992036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991307+0000","last_change":"2024-03-23T14:35:12.781319+0000","last_active":"2024-03-23T14:35:14.991307+0000","last_peered":"2024-03-23T14:35:14.991307+0000","last_clean":"2024-03-23T14:35:14.991307+0000","last_became_active":"2024-03-23T14:35:12.780393+0000","last_became_peered":"2024-03-23T14:35:12.780393+0000","last_unstale":"2024-03-23T14:35:14.991307+0000","last_undegraded":"2024-03-23T14:35:14.991307+0000","last_fullsized":"2024-03-23T14:35:14.991307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991191+0000","last_change":"2024-03-23T14:35:13.783755+0000","last_active":"2024-03-23T14:35:14.991191+0000","last_peered":"2024-03-23T14:35:14.991191+0000","last_clean":"2024-03-23T14:35:14.991191+0000","last_became_active":"2024-03-23T14:35:13.783415+0000","last_became_peered":"2024-03-23T14:35:13.783415+0000","last_unstale":"2024-03-23T14:35:14.991191+0000","last_undegraded":"2024-03-23T14:35:14.991191+0000","last_fullsized":"2024-03-23T14:35:14.991191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:07:11.094535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771472+0000","last_change":"2024-03-23T14:35:12.769935+0000","last_active":"2024-03-23T14:35:14.771472+0000","last_peered":"2024-03-23T14:35:14.771472+0000","last_clean":"2024-03-23T14:35:14.771472+0000","last_became_active":"2024-03-23T14:35:12.769638+0000","last_became_peered":"2024-03-23T14:35:12.769638+0000","last_unstale":"2024-03-23T14:35:14.771472+0000","last_undegraded":"2024-03-23T14:35:14.771472+0000","last_fullsized":"2024-03-23T14:35:14.771472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774313+0000","last_change":"2024-03-23T14:35:13.776295+0000","last_active":"2024-03-23T14:35:14.774313+0000","last_peered":"2024-03-23T14:35:14.774313+0000","last_clean":"2024-03-23T14:35:14.774313+0000","last_became_active":"2024-03-23T14:35:13.775968+0000","last_became_peered":"2024-03-23T14:35:13.775968+0000","last_unstale":"2024-03-23T14:35:14.774313+0000","last_undegraded":"2024-03-23T14:35:14.774313+0000","last_fullsized":"2024-03-23T14:35:14.774313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991315+0000","last_change":"2024-03-23T14:35:12.776276+0000","last_active":"2024-03-23T14:35:14.991315+0000","last_peered":"2024-03-23T14:35:14.991315+0000","last_clean":"2024-03-23T14:35:14.991315+0000","last_became_active":"2024-03-23T14:35:12.775990+0000","last_became_peered":"2024-03-23T14:35:12.775990+0000","last_unstale":"2024-03-23T14:35:14.991315+0000","last_undegraded":"2024-03-23T14:35:14.991315+0000","last_fullsized":"2024-03-23T14:35:14.991315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775094+0000","last_change":"2024-03-23T14:35:13.786554+0000","last_active":"2024-03-23T14:35:14.775094+0000","last_peered":"2024-03-23T14:35:14.775094+0000","last_clean":"2024-03-23T14:35:14.775094+0000","last_became_active":"2024-03-23T14:35:13.786286+0000","last_became_peered":"2024-03-23T14:35:13.786286+0000","last_unstale":"2024-03-23T14:35:14.775094+0000","last_undegraded":"2024-03-23T14:35:14.775094+0000","last_fullsized":"2024-03-23T14:35:14.775094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:40.665771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774420+0000","last_change":"2024-03-23T14:35:12.764502+0000","last_active":"2024-03-23T14:35:14.774420+0000","last_peered":"2024-03-23T14:35:14.774420+0000","last_clean":"2024-03-23T14:35:14.774420+0000","last_became_active":"2024-03-23T14:35:12.764241+0000","last_became_peered":"2024-03-23T14:35:12.764241+0000","last_unstale":"2024-03-23T14:35:14.774420+0000","last_undegraded":"2024-03-23T14:35:14.774420+0000","last_fullsized":"2024-03-23T14:35:14.774420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991506+0000","last_change":"2024-03-23T14:35:13.775872+0000","last_active":"2024-03-23T14:35:14.991506+0000","last_peered":"2024-03-23T14:35:14.991506+0000","last_clean":"2024-03-23T14:35:14.991506+0000","last_became_active":"2024-03-23T14:35:13.775527+0000","last_became_peered":"2024-03-23T14:35:13.775527+0000","last_unstale":"2024-03-23T14:35:14.991506+0000","last_undegraded":"2024-03-23T14:35:14.991506+0000","last_fullsized":"2024-03-23T14:35:14.991506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:17:27.919253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992153+0000","last_change":"2024-03-23T14:35:12.780722+0000","last_active":"2024-03-23T14:35:14.992153+0000","last_peered":"2024-03-23T14:35:14.992153+0000","last_clean":"2024-03-23T14:35:14.992153+0000","last_became_active":"2024-03-23T14:35:12.780349+0000","last_became_peered":"2024-03-23T14:35:12.780349+0000","last_unstale":"2024-03-23T14:35:14.992153+0000","last_undegraded":"2024-03-23T14:35:14.992153+0000","last_fullsized":"2024-03-23T14:35:14.992153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:50.822404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774384+0000","last_change":"2024-03-23T14:35:13.776300+0000","last_active":"2024-03-23T14:35:14.774384+0000","last_peered":"2024-03-23T14:35:14.774384+0000","last_clean":"2024-03-23T14:35:14.774384+0000","last_became_active":"2024-03-23T14:35:13.776007+0000","last_became_peered":"2024-03-23T14:35:13.776007+0000","last_unstale":"2024-03-23T14:35:14.774384+0000","last_undegraded":"2024-03-23T14:35:14.774384+0000","last_fullsized":"2024-03-23T14:35:14.774384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774585+0000","last_change":"2024-03-23T14:35:12.772616+0000","last_active":"2024-03-23T14:35:14.774585+0000","last_peered":"2024-03-23T14:35:14.774585+0000","last_clean":"2024-03-23T14:35:14.774585+0000","last_became_active":"2024-03-23T14:35:12.772237+0000","last_became_peered":"2024-03-23T14:35:12.772237+0000","last_unstale":"2024-03-23T14:35:14.774585+0000","last_undegraded":"2024-03-23T14:35:14.774585+0000","last_fullsized":"2024-03-23T14:35:14.774585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:50.822404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992233+0000","last_change":"2024-03-23T14:35:13.782070+0000","last_active":"2024-03-23T14:35:14.992233+0000","last_peered":"2024-03-23T14:35:14.992233+0000","last_clean":"2024-03-23T14:35:14.992233+0000","last_became_active":"2024-03-23T14:35:13.781798+0000","last_became_peered":"2024-03-23T14:35:13.781798+0000","last_unstale":"2024-03-23T14:35:14.992233+0000","last_undegraded":"2024-03-23T14:35:14.992233+0000","last_fullsized":"2024-03-23T14:35:14.992233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916782+0000","last_change":"2024-03-23T14:35:12.777518+0000","last_active":"2024-03-23T14:35:14.916782+0000","last_peered":"2024-03-23T14:35:14.916782+0000","last_clean":"2024-03-23T14:35:14.916782+0000","last_became_active":"2024-03-23T14:35:12.777242+0000","last_became_peered":"2024-03-23T14:35:12.777242+0000","last_unstale":"2024-03-23T14:35:14.916782+0000","last_undegraded":"2024-03-23T14:35:14.916782+0000","last_fullsized":"2024-03-23T14:35:14.916782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991622+0000","last_change":"2024-03-23T14:35:13.786237+0000","last_active":"2024-03-23T14:35:14.991622+0000","last_peered":"2024-03-23T14:35:14.991622+0000","last_clean":"2024-03-23T14:35:14.991622+0000","last_became_active":"2024-03-23T14:35:13.785920+0000","last_became_peered":"2024-03-23T14:35:13.785920+0000","last_unstale":"2024-03-23T14:35:14.991622+0000","last_undegraded":"2024-03-23T14:35:14.991622+0000","last_fullsized":"2024-03-23T14:35:14.991622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991213+0000","last_change":"2024-03-23T14:35:12.781097+0000","last_active":"2024-03-23T14:35:14.991213+0000","last_peered":"2024-03-23T14:35:14.991213+0000","last_clean":"2024-03-23T14:35:14.991213+0000","last_became_active":"2024-03-23T14:35:12.780408+0000","last_became_peered":"2024-03-23T14:35:12.780408+0000","last_unstale":"2024-03-23T14:35:14.991213+0000","last_undegraded":"2024-03-23T14:35:14.991213+0000","last_fullsized":"2024-03-23T14:35:14.991213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774978+0000","last_change":"2024-03-23T14:35:13.777707+0000","last_active":"2024-03-23T14:35:14.774978+0000","last_peered":"2024-03-23T14:35:14.774978+0000","last_clean":"2024-03-23T14:35:14.774978+0000","last_became_active":"2024-03-23T14:35:13.777470+0000","last_became_peered":"2024-03-23T14:35:13.777470+0000","last_unstale":"2024-03-23T14:35:14.774978+0000","last_undegraded":"2024-03-23T14:35:14.774978+0000","last_fullsized":"2024-03-23T14:35:14.774978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:39.242232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774400+0000","last_change":"2024-03-23T14:35:12.771937+0000","last_active":"2024-03-23T14:35:14.774400+0000","last_peered":"2024-03-23T14:35:14.774400+0000","last_clean":"2024-03-23T14:35:14.774400+0000","last_became_active":"2024-03-23T14:35:12.771560+0000","last_became_peered":"2024-03-23T14:35:12.771560+0000","last_unstale":"2024-03-23T14:35:14.774400+0000","last_undegraded":"2024-03-23T14:35:14.774400+0000","last_fullsized":"2024-03-23T14:35:14.774400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991517+0000","last_change":"2024-03-23T14:35:13.779913+0000","last_active":"2024-03-23T14:35:14.991517+0000","last_peered":"2024-03-23T14:35:14.991517+0000","last_clean":"2024-03-23T14:35:14.991517+0000","last_became_active":"2024-03-23T14:35:13.779667+0000","last_became_peered":"2024-03-23T14:35:13.779667+0000","last_unstale":"2024-03-23T14:35:14.991517+0000","last_undegraded":"2024-03-23T14:35:14.991517+0000","last_fullsized":"2024-03-23T14:35:14.991517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774464+0000","last_change":"2024-03-23T14:35:12.770228+0000","last_active":"2024-03-23T14:35:14.774464+0000","last_peered":"2024-03-23T14:35:14.774464+0000","last_clean":"2024-03-23T14:35:14.774464+0000","last_became_active":"2024-03-23T14:35:12.769962+0000","last_became_peered":"2024-03-23T14:35:12.769962+0000","last_unstale":"2024-03-23T14:35:14.774464+0000","last_undegraded":"2024-03-23T14:35:14.774464+0000","last_fullsized":"2024-03-23T14:35:14.774464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991918+0000","last_change":"2024-03-23T14:35:13.786710+0000","last_active":"2024-03-23T14:35:14.991918+0000","last_peered":"2024-03-23T14:35:14.991918+0000","last_clean":"2024-03-23T14:35:14.991918+0000","last_became_active":"2024-03-23T14:35:13.786454+0000","last_became_peered":"2024-03-23T14:35:13.786454+0000","last_unstale":"2024-03-23T14:35:14.991918+0000","last_undegraded":"2024-03-23T14:35:14.991918+0000","last_fullsized":"2024-03-23T14:35:14.991918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992224+0000","last_change":"2024-03-23T14:35:12.778699+0000","last_active":"2024-03-23T14:35:14.992224+0000","last_peered":"2024-03-23T14:35:14.992224+0000","last_clean":"2024-03-23T14:35:14.992224+0000","last_became_active":"2024-03-23T14:35:12.778416+0000","last_became_peered":"2024-03-23T14:35:12.778416+0000","last_unstale":"2024-03-23T14:35:14.992224+0000","last_undegraded":"2024-03-23T14:35:14.992224+0000","last_fullsized":"2024-03-23T14:35:14.992224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916656+0000","last_change":"2024-03-23T14:35:13.773983+0000","last_active":"2024-03-23T14:35:14.916656+0000","last_peered":"2024-03-23T14:35:14.916656+0000","last_clean":"2024-03-23T14:35:14.916656+0000","last_became_active":"2024-03-23T14:35:13.773689+0000","last_became_peered":"2024-03-23T14:35:13.773689+0000","last_unstale":"2024-03-23T14:35:14.916656+0000","last_undegraded":"2024-03-23T14:35:14.916656+0000","last_fullsized":"2024-03-23T14:35:14.916656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991352+0000","last_change":"2024-03-23T14:35:13.783572+0000","last_active":"2024-03-23T14:35:14.991352+0000","last_peered":"2024-03-23T14:35:14.991352+0000","last_clean":"2024-03-23T14:35:14.991352+0000","last_became_active":"2024-03-23T14:35:13.782224+0000","last_became_peered":"2024-03-23T14:35:13.782224+0000","last_unstale":"2024-03-23T14:35:14.991352+0000","last_undegraded":"2024-03-23T14:35:14.991352+0000","last_fullsized":"2024-03-23T14:35:14.991352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:20.569093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991763+0000","last_change":"2024-03-23T14:35:12.775487+0000","last_active":"2024-03-23T14:35:14.991763+0000","last_peered":"2024-03-23T14:35:14.991763+0000","last_clean":"2024-03-23T14:35:14.991763+0000","last_became_active":"2024-03-23T14:35:12.775208+0000","last_became_peered":"2024-03-23T14:35:12.775208+0000","last_unstale":"2024-03-23T14:35:14.991763+0000","last_undegraded":"2024-03-23T14:35:14.991763+0000","last_fullsized":"2024-03-23T14:35:14.991763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991431+0000","last_change":"2024-03-23T14:35:13.783879+0000","last_active":"2024-03-23T14:35:14.991431+0000","last_peered":"2024-03-23T14:35:14.991431+0000","last_clean":"2024-03-23T14:35:14.991431+0000","last_became_active":"2024-03-23T14:35:13.783451+0000","last_became_peered":"2024-03-23T14:35:13.783451+0000","last_unstale":"2024-03-23T14:35:14.991431+0000","last_undegraded":"2024-03-23T14:35:14.991431+0000","last_fullsized":"2024-03-23T14:35:14.991431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:54:02.413325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992060+0000","last_change":"2024-03-23T14:35:12.777520+0000","last_active":"2024-03-23T14:35:14.992060+0000","last_peered":"2024-03-23T14:35:14.992060+0000","last_clean":"2024-03-23T14:35:14.992060+0000","last_became_active":"2024-03-23T14:35:12.777254+0000","last_became_peered":"2024-03-23T14:35:12.777254+0000","last_unstale":"2024-03-23T14:35:14.992060+0000","last_undegraded":"2024-03-23T14:35:14.992060+0000","last_fullsized":"2024-03-23T14:35:14.992060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:43.686652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991341+0000","last_change":"2024-03-23T14:35:13.785243+0000","last_active":"2024-03-23T14:35:14.991341+0000","last_peered":"2024-03-23T14:35:14.991341+0000","last_clean":"2024-03-23T14:35:14.991341+0000","last_became_active":"2024-03-23T14:35:13.784972+0000","last_became_peered":"2024-03-23T14:35:13.784972+0000","last_unstale":"2024-03-23T14:35:14.991341+0000","last_undegraded":"2024-03-23T14:35:14.991341+0000","last_fullsized":"2024-03-23T14:35:14.991341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:24:06.436157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992114+0000","last_change":"2024-03-23T14:35:12.770618+0000","last_active":"2024-03-23T14:35:14.992114+0000","last_peered":"2024-03-23T14:35:14.992114+0000","last_clean":"2024-03-23T14:35:14.992114+0000","last_became_active":"2024-03-23T14:35:12.770320+0000","last_became_peered":"2024-03-23T14:35:12.770320+0000","last_unstale":"2024-03-23T14:35:14.992114+0000","last_undegraded":"2024-03-23T14:35:14.992114+0000","last_fullsized":"2024-03-23T14:35:14.992114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:33:10.353085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991383+0000","last_change":"2024-03-23T14:35:13.775981+0000","last_active":"2024-03-23T14:35:14.991383+0000","last_peered":"2024-03-23T14:35:14.991383+0000","last_clean":"2024-03-23T14:35:14.991383+0000","last_became_active":"2024-03-23T14:35:13.775695+0000","last_became_peered":"2024-03-23T14:35:13.775695+0000","last_unstale":"2024-03-23T14:35:14.991383+0000","last_undegraded":"2024-03-23T14:35:14.991383+0000","last_fullsized":"2024-03-23T14:35:14.991383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991901+0000","last_change":"2024-03-23T14:35:12.773386+0000","last_active":"2024-03-23T14:35:14.991901+0000","last_peered":"2024-03-23T14:35:14.991901+0000","last_clean":"2024-03-23T14:35:14.991901+0000","last_became_active":"2024-03-23T14:35:12.773104+0000","last_became_peered":"2024-03-23T14:35:12.773104+0000","last_unstale":"2024-03-23T14:35:14.991901+0000","last_undegraded":"2024-03-23T14:35:14.991901+0000","last_fullsized":"2024-03-23T14:35:14.991901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:34.363745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":196608,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":64794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:35:16.731 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:35:16.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.872+0000 7fb08ea15640 1 Processor -- start 2024-03-23T14:35:16.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.872+0000 7fb08ea15640 1 -- start start 2024-03-23T14:35:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb08ea15640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb0880ad850 0x7fb0880cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb08ea15640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb08802c1c0 con 0x7fb0880cf840 2024-03-23T14:35:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb08ea15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb088071630 con 0x7fb0880cd380 2024-03-23T14:35:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb08ea15640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb08802c650 con 0x7fb0880ad850 2024-03-23T14:35:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb0877fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb0880ad850 0x7fb0880cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb0877fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb0880ad850 0x7fb0880cb910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41022/0 (socket says 172.21.15.161:41022) 2024-03-23T14:35:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb0877fe640 1 -- 172.21.15.161:0/3295472340 learned_addr learned my addr 172.21.15.161:0/3295472340 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb087fff640 1 --1- 172.21.15.161:0/3295472340 >> v1:172.21.15.161:6789/0 conn(0x7fb0880cf840 0x7fb0880ad140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49822/0 (socket says 172.21.15.161:49822) 2024-03-23T14:35:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb0877fe640 1 -- 172.21.15.161:0/3295472340 >> v1:172.21.15.183:6789/0 conn(0x7fb0880cd380 legacy=0x7fb088168810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb0877fe640 1 -- 172.21.15.161:0/3295472340 >> v1:172.21.15.161:6789/0 conn(0x7fb0880cf840 legacy=0x7fb0880ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb086ffd640 1 -- 172.21.15.161:0/3295472340 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4149954129 0 0) 0x7fb08802c1c0 con 0x7fb0880cf840 2024-03-23T14:35:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.873+0000 7fb0877fe640 1 -- 172.21.15.161:0/3295472340 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0880af490 con 0x7fb0880ad850 2024-03-23T14:35:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.874+0000 7fb0877fe640 1 --2- 172.21.15.161:0/3295472340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880ad850 0x7fb0880cb910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009870 tx=0x7fb07c02ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387b000ec4966e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.874+0000 7fb086ffd640 1 -- 172.21.15.161:0/3295472340 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07c02f840 con 0x7fb0880ad850 2024-03-23T14:35:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.874+0000 7fb086ffd640 1 -- 172.21.15.161:0/3295472340 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb07c02f9a0 con 0x7fb0880ad850 2024-03-23T14:35:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.874+0000 7fb086ffd640 1 -- 172.21.15.161:0/3295472340 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07c02fd30 con 0x7fb0880ad850 2024-03-23T14:35:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.875+0000 7fb08ea15640 1 -- 172.21.15.161:0/3295472340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880ad850 msgr2=0x7fb0880cb910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.875+0000 7fb08ea15640 1 --2- 172.21.15.161:0/3295472340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880ad850 0x7fb0880cb910 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb07c009870 tx=0x7fb07c02ba20 comp rx=0 tx=0).stop 2024-03-23T14:35:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.875+0000 7fb08ea15640 1 -- 172.21.15.161:0/3295472340 shutdown_connections 2024-03-23T14:35:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.875+0000 7fb08ea15640 1 --2- 172.21.15.161:0/3295472340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880ad850 0x7fb0880cb910 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.875+0000 7fb08ea15640 1 -- 172.21.15.161:0/3295472340 >> 172.21.15.161:0/3295472340 conn(0x7fb0880b1e60 msgr2=0x7fb0880b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:16.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.875+0000 7fb08ea15640 1 -- 172.21.15.161:0/3295472340 shutdown_connections 2024-03-23T14:35:16.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.875+0000 7fb08ea15640 1 -- 172.21.15.161:0/3295472340 wait complete. 2024-03-23T14:35:16.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.876+0000 7fb08ea15640 1 Processor -- start 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.876+0000 7fb08ea15640 1 -- start start 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb08ea15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0880ad850 0x7fb0880c7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb08ea15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880cd380 0x7fb088157ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb08ea15640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0880cf840 0x7fb0880c45b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb08ea15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880b4d50 con 0x7fb0880cd380 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb08ea15640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb0880c78c0 con 0x7fb0880cf840 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb08ea15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb0880b2a90 con 0x7fb0880ad850 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880cd380 0x7fb088157ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880cd380 0x7fb088157ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41026/0 (socket says 172.21.15.161:41026) 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 -- 172.21.15.161:0/3638839890 learned_addr learned my addr 172.21.15.161:0/3638839890 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:16.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb087fff640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0880ad850 0x7fb0880c7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb08cfb2640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0880cf840 0x7fb0880c45b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 -- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0880ad850 msgr2=0x7fb0880c7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0880ad850 0x7fb0880c7380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 -- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0880cf840 msgr2=0x7fb0880c45b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0880cf840 0x7fb0880c45b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 -- 172.21.15.161:0/3638839890 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb07c0094d0 con 0x7fb0880cd380 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb0877fe640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880cd380 0x7fb088157ab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb07c02ba60 tx=0x7fb07c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb084ff9640 1 -- 172.21.15.161:0/3638839890 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07c02f810 con 0x7fb0880cd380 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb084ff9640 1 -- 172.21.15.161:0/3638839890 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb07c042480 con 0x7fb0880cd380 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb084ff9640 1 -- 172.21.15.161:0/3638839890 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb07c042850 con 0x7fb0880cd380 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.877+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0880b4140 con 0x7fb0880cd380 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.878+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0880b4700 con 0x7fb0880cd380 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.879+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb08809e4c0 con 0x7fb0880cd380 2024-03-23T14:35:16.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.879+0000 7fb084ff9640 1 -- 172.21.15.161:0/3638839890 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb07c02d3d0 con 0x7fb0880cd380 2024-03-23T14:35:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.879+0000 7fb084ff9640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb07406b810 0x7fb07406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.879+0000 7fb084ff9640 1 -- 172.21.15.161:0/3638839890 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb07c03c070 con 0x7fb0880cd380 2024-03-23T14:35:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.879+0000 7fb087fff640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb07406b810 0x7fb07406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.880+0000 7fb087fff640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb07406b810 0x7fb07406dcd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb078007f70 tx=0x7fb078007f00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:16.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:16.884+0000 7fb084ff9640 1 -- 172.21.15.161:0/3638839890 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb07c0774d0 con 0x7fb0880cd380 2024-03-23T14:35:17.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.052+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb0880cbe50 con 0x7fb07406b810 2024-03-23T14:35:17.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.058+0000 7fb084ff9640 1 -- 172.21.15.161:0/3638839890 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251945 (secure 0 0 0) 0x7fb078004c70 con 0x7fb07406b810 2024-03-23T14:35:17.057 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:17.058 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb07406b810 msgr2=0x7fb07406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb07406b810 0x7fb07406dcd0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb078007f70 tx=0x7fb078007f00 comp rx=0 tx=0).stop 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880cd380 msgr2=0x7fb088157ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880cd380 0x7fb088157ab0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb07c02ba60 tx=0x7fb07c004730 comp rx=0 tx=0).stop 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 shutdown_connections 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb07406b810 0x7fb07406dcd0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb0880cf840 0x7fb0880c45b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb0880cd380 0x7fb088157ab0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 --2- 172.21.15.161:0/3638839890 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb0880ad850 0x7fb0880c7380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 >> 172.21.15.161:0/3638839890 conn(0x7fb0880b1e60 msgr2=0x7fb0880d08f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:17.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 shutdown_connections 2024-03-23T14:35:17.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.061+0000 7fb08ea15640 1 -- 172.21.15.161:0/3638839890 wait complete. 2024-03-23T14:35:17.068 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-03-23T14:35:16.474840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":203,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":141,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219016,"kb_used_data":2800,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755704,"statfs":{"total":773094113280,"available":772869840896,"internally_reserved":0,"allocated":2867200,"data_stored":1740428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":30,"apply_latency_ms":30,"commit_latency_ns":30000000,"apply_latency_ns":30000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.728276"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991991+0000","last_change":"2024-03-23T14:35:13.778548+0000","last_active":"2024-03-23T14:35:14.991991+0000","last_peered":"2024-03-23T14:35:14.991991+0000","last_clean":"2024-03-23T14:35:14.991991+0000","last_became_active":"2024-03-23T14:35:13.778166+0000","last_became_peered":"2024-03-23T14:35:13.778166+0000","last_unstale":"2024-03-23T14:35:14.991991+0000","last_undegraded":"2024-03-23T14:35:14.991991+0000","last_fullsized":"2024-03-23T14:35:14.991991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:19.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774593+0000","last_change":"2024-03-23T14:35:13.788107+0000","last_active":"2024-03-23T14:35:14.774593+0000","last_peered":"2024-03-23T14:35:14.774593+0000","last_clean":"2024-03-23T14:35:14.774593+0000","last_became_active":"2024-03-23T14:35:13.787705+0000","last_became_peered":"2024-03-23T14:35:13.787705+0000","last_unstale":"2024-03-23T14:35:14.774593+0000","last_undegraded":"2024-03-23T14:35:14.774593+0000","last_fullsized":"2024-03-23T14:35:14.774593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991567+0000","last_change":"2024-03-23T14:35:13.783735+0000","last_active":"2024-03-23T14:35:14.991567+0000","last_peered":"2024-03-23T14:35:14.991567+0000","last_clean":"2024-03-23T14:35:14.991567+0000","last_became_active":"2024-03-23T14:35:13.783349+0000","last_became_peered":"2024-03-23T14:35:13.783349+0000","last_unstale":"2024-03-23T14:35:14.991567+0000","last_undegraded":"2024-03-23T14:35:14.991567+0000","last_fullsized":"2024-03-23T14:35:14.991567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991261+0000","last_change":"2024-03-23T14:35:13.787909+0000","last_active":"2024-03-23T14:35:14.991261+0000","last_peered":"2024-03-23T14:35:14.991261+0000","last_clean":"2024-03-23T14:35:14.991261+0000","last_became_active":"2024-03-23T14:35:13.786041+0000","last_became_peered":"2024-03-23T14:35:13.786041+0000","last_unstale":"2024-03-23T14:35:14.991261+0000","last_undegraded":"2024-03-23T14:35:14.991261+0000","last_fullsized":"2024-03-23T14:35:14.991261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991979+0000","last_change":"2024-03-23T14:35:13.784230+0000","last_active":"2024-03-23T14:35:14.991979+0000","last_peered":"2024-03-23T14:35:14.991979+0000","last_clean":"2024-03-23T14:35:14.991979+0000","last_became_active":"2024-03-23T14:35:13.781446+0000","last_became_peered":"2024-03-23T14:35:13.781446+0000","last_unstale":"2024-03-23T14:35:14.991979+0000","last_undegraded":"2024-03-23T14:35:14.991979+0000","last_fullsized":"2024-03-23T14:35:14.991979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:20.569093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991109+0000","last_change":"2024-03-23T14:35:13.785787+0000","last_active":"2024-03-23T14:35:14.991109+0000","last_peered":"2024-03-23T14:35:14.991109+0000","last_clean":"2024-03-23T14:35:14.991109+0000","last_became_active":"2024-03-23T14:35:13.785528+0000","last_became_peered":"2024-03-23T14:35:13.785528+0000","last_unstale":"2024-03-23T14:35:14.991109+0000","last_undegraded":"2024-03-23T14:35:14.991109+0000","last_fullsized":"2024-03-23T14:35:14.991109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991672+0000","last_change":"2024-03-23T14:35:13.787315+0000","last_active":"2024-03-23T14:35:14.991672+0000","last_peered":"2024-03-23T14:35:14.991672+0000","last_clean":"2024-03-23T14:35:14.991672+0000","last_became_active":"2024-03-23T14:35:13.787032+0000","last_became_peered":"2024-03-23T14:35:13.787032+0000","last_unstale":"2024-03-23T14:35:14.991672+0000","last_undegraded":"2024-03-23T14:35:14.991672+0000","last_fullsized":"2024-03-23T14:35:14.991672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991225+0000","last_change":"2024-03-23T14:35:13.779324+0000","last_active":"2024-03-23T14:35:14.991225+0000","last_peered":"2024-03-23T14:35:14.991225+0000","last_clean":"2024-03-23T14:35:14.991225+0000","last_became_active":"2024-03-23T14:35:13.779048+0000","last_became_peered":"2024-03-23T14:35:13.779048+0000","last_unstale":"2024-03-23T14:35:14.991225+0000","last_undegraded":"2024-03-23T14:35:14.991225+0000","last_fullsized":"2024-03-23T14:35:14.991225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:37.554855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992038+0000","last_change":"2024-03-23T14:35:13.781031+0000","last_active":"2024-03-23T14:35:14.992038+0000","last_peered":"2024-03-23T14:35:14.992038+0000","last_clean":"2024-03-23T14:35:14.992038+0000","last_became_active":"2024-03-23T14:35:13.780762+0000","last_became_peered":"2024-03-23T14:35:13.780762+0000","last_unstale":"2024-03-23T14:35:14.992038+0000","last_undegraded":"2024-03-23T14:35:14.992038+0000","last_fullsized":"2024-03-23T14:35:14.992038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:17:27.919253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916812+0000","last_change":"2024-03-23T14:35:13.776904+0000","last_active":"2024-03-23T14:35:14.916812+0000","last_peered":"2024-03-23T14:35:14.916812+0000","last_clean":"2024-03-23T14:35:14.916812+0000","last_became_active":"2024-03-23T14:35:13.776499+0000","last_became_peered":"2024-03-23T14:35:13.776499+0000","last_unstale":"2024-03-23T14:35:14.916812+0000","last_undegraded":"2024-03-23T14:35:14.916812+0000","last_fullsized":"2024-03-23T14:35:14.916812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916738+0000","last_change":"2024-03-23T14:35:13.777465+0000","last_active":"2024-03-23T14:35:14.916738+0000","last_peered":"2024-03-23T14:35:14.916738+0000","last_clean":"2024-03-23T14:35:14.916738+0000","last_became_active":"2024-03-23T14:35:13.777169+0000","last_became_peered":"2024-03-23T14:35:13.777169+0000","last_unstale":"2024-03-23T14:35:14.916738+0000","last_undegraded":"2024-03-23T14:35:14.916738+0000","last_fullsized":"2024-03-23T14:35:14.916738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775225+0000","last_change":"2024-03-23T14:35:13.776334+0000","last_active":"2024-03-23T14:35:14.775225+0000","last_peered":"2024-03-23T14:35:14.775225+0000","last_clean":"2024-03-23T14:35:14.775225+0000","last_became_active":"2024-03-23T14:35:13.776069+0000","last_became_peered":"2024-03-23T14:35:13.776069+0000","last_unstale":"2024-03-23T14:35:14.775225+0000","last_undegraded":"2024-03-23T14:35:14.775225+0000","last_fullsized":"2024-03-23T14:35:14.775225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:44.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991775+0000","last_change":"2024-03-23T14:35:13.779105+0000","last_active":"2024-03-23T14:35:14.991775+0000","last_peered":"2024-03-23T14:35:14.991775+0000","last_clean":"2024-03-23T14:35:14.991775+0000","last_became_active":"2024-03-23T14:35:13.778836+0000","last_became_peered":"2024-03-23T14:35:13.778836+0000","last_unstale":"2024-03-23T14:35:14.991775+0000","last_undegraded":"2024-03-23T14:35:14.991775+0000","last_fullsized":"2024-03-23T14:35:14.991775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991202+0000","last_change":"2024-03-23T14:35:13.786094+0000","last_active":"2024-03-23T14:35:14.991202+0000","last_peered":"2024-03-23T14:35:14.991202+0000","last_clean":"2024-03-23T14:35:14.991202+0000","last_became_active":"2024-03-23T14:35:13.785749+0000","last_became_peered":"2024-03-23T14:35:13.785749+0000","last_unstale":"2024-03-23T14:35:14.991202+0000","last_undegraded":"2024-03-23T14:35:14.991202+0000","last_fullsized":"2024-03-23T14:35:14.991202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:48:45.763326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774304+0000","last_change":"2024-03-23T14:35:13.763672+0000","last_active":"2024-03-23T14:35:14.774304+0000","last_peered":"2024-03-23T14:35:14.774304+0000","last_clean":"2024-03-23T14:35:14.774304+0000","last_became_active":"2024-03-23T14:35:13.763446+0000","last_became_peered":"2024-03-23T14:35:13.763446+0000","last_unstale":"2024-03-23T14:35:14.774304+0000","last_undegraded":"2024-03-23T14:35:14.774304+0000","last_fullsized":"2024-03-23T14:35:14.774304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.990985+0000","last_change":"2024-03-23T14:35:13.788261+0000","last_active":"2024-03-23T14:35:14.990985+0000","last_peered":"2024-03-23T14:35:14.990985+0000","last_clean":"2024-03-23T14:35:14.990985+0000","last_became_active":"2024-03-23T14:35:13.787931+0000","last_became_peered":"2024-03-23T14:35:13.787931+0000","last_unstale":"2024-03-23T14:35:14.990985+0000","last_undegraded":"2024-03-23T14:35:14.990985+0000","last_fullsized":"2024-03-23T14:35:14.990985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991568+0000","last_change":"2024-03-23T14:35:13.788504+0000","last_active":"2024-03-23T14:35:14.991568+0000","last_peered":"2024-03-23T14:35:14.991568+0000","last_clean":"2024-03-23T14:35:14.991568+0000","last_became_active":"2024-03-23T14:35:13.788220+0000","last_became_peered":"2024-03-23T14:35:13.788220+0000","last_unstale":"2024-03-23T14:35:14.991568+0000","last_undegraded":"2024-03-23T14:35:14.991568+0000","last_fullsized":"2024-03-23T14:35:14.991568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:25.033547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771552+0000","last_change":"2024-03-23T14:35:13.777872+0000","last_active":"2024-03-23T14:35:14.771552+0000","last_peered":"2024-03-23T14:35:14.771552+0000","last_clean":"2024-03-23T14:35:14.771552+0000","last_became_active":"2024-03-23T14:35:13.777618+0000","last_became_peered":"2024-03-23T14:35:13.777618+0000","last_unstale":"2024-03-23T14:35:14.771552+0000","last_undegraded":"2024-03-23T14:35:14.771552+0000","last_fullsized":"2024-03-23T14:35:14.771552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:51.829515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992132+0000","last_change":"2024-03-23T14:35:13.785636+0000","last_active":"2024-03-23T14:35:14.992132+0000","last_peered":"2024-03-23T14:35:14.992132+0000","last_clean":"2024-03-23T14:35:14.992132+0000","last_became_active":"2024-03-23T14:35:13.784525+0000","last_became_peered":"2024-03-23T14:35:13.784525+0000","last_unstale":"2024-03-23T14:35:14.992132+0000","last_undegraded":"2024-03-23T14:35:14.992132+0000","last_fullsized":"2024-03-23T14:35:14.992132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:25.033547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916605+0000","last_change":"2024-03-23T14:35:13.775546+0000","last_active":"2024-03-23T14:35:14.916605+0000","last_peered":"2024-03-23T14:35:14.916605+0000","last_clean":"2024-03-23T14:35:14.916605+0000","last_became_active":"2024-03-23T14:35:13.774705+0000","last_became_peered":"2024-03-23T14:35:13.774705+0000","last_unstale":"2024-03-23T14:35:14.916605+0000","last_undegraded":"2024-03-23T14:35:14.916605+0000","last_fullsized":"2024-03-23T14:35:14.916605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991370+0000","last_change":"2024-03-23T14:35:13.785890+0000","last_active":"2024-03-23T14:35:14.991370+0000","last_peered":"2024-03-23T14:35:14.991370+0000","last_clean":"2024-03-23T14:35:14.991370+0000","last_became_active":"2024-03-23T14:35:13.785564+0000","last_became_peered":"2024-03-23T14:35:13.785564+0000","last_unstale":"2024-03-23T14:35:14.991370+0000","last_undegraded":"2024-03-23T14:35:14.991370+0000","last_fullsized":"2024-03-23T14:35:14.991370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:16:34.110338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991141+0000","last_change":"2024-03-23T14:35:13.785106+0000","last_active":"2024-03-23T14:35:14.991141+0000","last_peered":"2024-03-23T14:35:14.991141+0000","last_clean":"2024-03-23T14:35:14.991141+0000","last_became_active":"2024-03-23T14:35:13.784543+0000","last_became_peered":"2024-03-23T14:35:13.784543+0000","last_unstale":"2024-03-23T14:35:14.991141+0000","last_undegraded":"2024-03-23T14:35:14.991141+0000","last_fullsized":"2024-03-23T14:35:14.991141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991371+0000","last_change":"2024-03-23T14:35:13.782489+0000","last_active":"2024-03-23T14:35:14.991371+0000","last_peered":"2024-03-23T14:35:14.991371+0000","last_clean":"2024-03-23T14:35:14.991371+0000","last_became_active":"2024-03-23T14:35:13.782292+0000","last_became_peered":"2024-03-23T14:35:13.782292+0000","last_unstale":"2024-03-23T14:35:14.991371+0000","last_undegraded":"2024-03-23T14:35:14.991371+0000","last_fullsized":"2024-03-23T14:35:14.991371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:16:34.110338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991174+0000","last_change":"2024-03-23T14:35:13.776346+0000","last_active":"2024-03-23T14:35:14.991174+0000","last_peered":"2024-03-23T14:35:14.991174+0000","last_clean":"2024-03-23T14:35:14.991174+0000","last_became_active":"2024-03-23T14:35:13.776085+0000","last_became_peered":"2024-03-23T14:35:13.776085+0000","last_unstale":"2024-03-23T14:35:14.991174+0000","last_undegraded":"2024-03-23T14:35:14.991174+0000","last_fullsized":"2024-03-23T14:35:14.991174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992164+0000","last_change":"2024-03-23T14:35:13.781666+0000","last_active":"2024-03-23T14:35:14.992164+0000","last_peered":"2024-03-23T14:35:14.992164+0000","last_clean":"2024-03-23T14:35:14.992164+0000","last_became_active":"2024-03-23T14:35:13.781380+0000","last_became_peered":"2024-03-23T14:35:13.781380+0000","last_unstale":"2024-03-23T14:35:14.992164+0000","last_undegraded":"2024-03-23T14:35:14.992164+0000","last_fullsized":"2024-03-23T14:35:14.992164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916837+0000","last_change":"2024-03-23T14:35:13.774246+0000","last_active":"2024-03-23T14:35:14.916837+0000","last_peered":"2024-03-23T14:35:14.916837+0000","last_clean":"2024-03-23T14:35:14.916837+0000","last_became_active":"2024-03-23T14:35:13.773945+0000","last_became_peered":"2024-03-23T14:35:13.773945+0000","last_unstale":"2024-03-23T14:35:14.916837+0000","last_undegraded":"2024-03-23T14:35:14.916837+0000","last_fullsized":"2024-03-23T14:35:14.916837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774384+0000","last_change":"2024-03-23T14:35:13.774154+0000","last_active":"2024-03-23T14:35:14.774384+0000","last_peered":"2024-03-23T14:35:14.774384+0000","last_clean":"2024-03-23T14:35:14.774384+0000","last_became_active":"2024-03-23T14:35:13.773320+0000","last_became_peered":"2024-03-23T14:35:13.773320+0000","last_unstale":"2024-03-23T14:35:14.774384+0000","last_undegraded":"2024-03-23T14:35:14.774384+0000","last_fullsized":"2024-03-23T14:35:14.774384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991890+0000","last_change":"2024-03-23T14:35:13.782131+0000","last_active":"2024-03-23T14:35:14.991890+0000","last_peered":"2024-03-23T14:35:14.991890+0000","last_clean":"2024-03-23T14:35:14.991890+0000","last_became_active":"2024-03-23T14:35:13.781455+0000","last_became_peered":"2024-03-23T14:35:13.781455+0000","last_unstale":"2024-03-23T14:35:14.991890+0000","last_undegraded":"2024-03-23T14:35:14.991890+0000","last_fullsized":"2024-03-23T14:35:14.991890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991510+0000","last_change":"2024-03-23T14:35:13.787101+0000","last_active":"2024-03-23T14:35:14.991510+0000","last_peered":"2024-03-23T14:35:14.991510+0000","last_clean":"2024-03-23T14:35:14.991510+0000","last_became_active":"2024-03-23T14:35:13.786742+0000","last_became_peered":"2024-03-23T14:35:13.786742+0000","last_unstale":"2024-03-23T14:35:14.991510+0000","last_undegraded":"2024-03-23T14:35:14.991510+0000","last_fullsized":"2024-03-23T14:35:14.991510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991510+0000","last_change":"2024-03-23T14:35:13.781923+0000","last_active":"2024-03-23T14:35:14.991510+0000","last_peered":"2024-03-23T14:35:14.991510+0000","last_clean":"2024-03-23T14:35:14.991510+0000","last_became_active":"2024-03-23T14:35:13.781553+0000","last_became_peered":"2024-03-23T14:35:13.781553+0000","last_unstale":"2024-03-23T14:35:14.991510+0000","last_undegraded":"2024-03-23T14:35:14.991510+0000","last_fullsized":"2024-03-23T14:35:14.991510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:31:20.667310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771511+0000","last_change":"2024-03-23T14:35:13.775171+0000","last_active":"2024-03-23T14:35:14.771511+0000","last_peered":"2024-03-23T14:35:14.771511+0000","last_clean":"2024-03-23T14:35:14.771511+0000","last_became_active":"2024-03-23T14:35:13.774898+0000","last_became_peered":"2024-03-23T14:35:13.774898+0000","last_unstale":"2024-03-23T14:35:14.771511+0000","last_undegraded":"2024-03-23T14:35:14.771511+0000","last_fullsized":"2024-03-23T14:35:14.771511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:44.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916695+0000","last_change":"2024-03-23T14:35:13.785730+0000","last_active":"2024-03-23T14:35:14.916695+0000","last_peered":"2024-03-23T14:35:14.916695+0000","last_clean":"2024-03-23T14:35:14.916695+0000","last_became_active":"2024-03-23T14:35:13.785427+0000","last_became_peered":"2024-03-23T14:35:13.785427+0000","last_unstale":"2024-03-23T14:35:14.916695+0000","last_undegraded":"2024-03-23T14:35:14.916695+0000","last_fullsized":"2024-03-23T14:35:14.916695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774946+0000","last_change":"2024-03-23T14:35:13.769227+0000","last_active":"2024-03-23T14:35:14.774946+0000","last_peered":"2024-03-23T14:35:14.774946+0000","last_clean":"2024-03-23T14:35:14.774946+0000","last_became_active":"2024-03-23T14:35:13.768866+0000","last_became_peered":"2024-03-23T14:35:13.768866+0000","last_unstale":"2024-03-23T14:35:14.774946+0000","last_undegraded":"2024-03-23T14:35:14.774946+0000","last_fullsized":"2024-03-23T14:35:14.774946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:26.689847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992031+0000","last_change":"2024-03-23T14:35:12.777511+0000","last_active":"2024-03-23T14:35:14.992031+0000","last_peered":"2024-03-23T14:35:14.992031+0000","last_clean":"2024-03-23T14:35:14.992031+0000","last_became_active":"2024-03-23T14:35:12.777173+0000","last_became_peered":"2024-03-23T14:35:12.777173+0000","last_unstale":"2024-03-23T14:35:14.992031+0000","last_undegraded":"2024-03-23T14:35:14.992031+0000","last_fullsized":"2024-03-23T14:35:14.992031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:50.295934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991649+0000","last_change":"2024-03-23T14:35:13.784656+0000","last_active":"2024-03-23T14:35:14.991649+0000","last_peered":"2024-03-23T14:35:14.991649+0000","last_clean":"2024-03-23T14:35:14.991649+0000","last_became_active":"2024-03-23T14:35:13.783372+0000","last_became_peered":"2024-03-23T14:35:13.783372+0000","last_unstale":"2024-03-23T14:35:14.991649+0000","last_undegraded":"2024-03-23T14:35:14.991649+0000","last_fullsized":"2024-03-23T14:35:14.991649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916556+0000","last_change":"2024-03-23T14:35:12.772860+0000","last_active":"2024-03-23T14:35:14.916556+0000","last_peered":"2024-03-23T14:35:14.916556+0000","last_clean":"2024-03-23T14:35:14.916556+0000","last_became_active":"2024-03-23T14:35:12.772437+0000","last_became_peered":"2024-03-23T14:35:12.772437+0000","last_unstale":"2024-03-23T14:35:14.916556+0000","last_undegraded":"2024-03-23T14:35:14.916556+0000","last_fullsized":"2024-03-23T14:35:14.916556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991525+0000","last_change":"2024-03-23T14:35:12.773627+0000","last_active":"2024-03-23T14:35:14.991525+0000","last_peered":"2024-03-23T14:35:14.991525+0000","last_clean":"2024-03-23T14:35:14.991525+0000","last_became_active":"2024-03-23T14:35:12.773323+0000","last_became_peered":"2024-03-23T14:35:12.773323+0000","last_unstale":"2024-03-23T14:35:14.991525+0000","last_undegraded":"2024-03-23T14:35:14.991525+0000","last_fullsized":"2024-03-23T14:35:14.991525+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991373+0000","last_change":"2024-03-23T14:35:13.786951+0000","last_active":"2024-03-23T14:35:14.991373+0000","last_peered":"2024-03-23T14:35:14.991373+0000","last_clean":"2024-03-23T14:35:14.991373+0000","last_became_active":"2024-03-23T14:35:13.786684+0000","last_became_peered":"2024-03-23T14:35:13.786684+0000","last_unstale":"2024-03-23T14:35:14.991373+0000","last_undegraded":"2024-03-23T14:35:14.991373+0000","last_fullsized":"2024-03-23T14:35:14.991373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991479+0000","last_change":"2024-03-23T14:35:12.775888+0000","last_active":"2024-03-23T14:35:14.991479+0000","last_peered":"2024-03-23T14:35:14.991479+0000","last_clean":"2024-03-23T14:35:14.991479+0000","last_became_active":"2024-03-23T14:35:12.775610+0000","last_became_peered":"2024-03-23T14:35:12.775610+0000","last_unstale":"2024-03-23T14:35:14.991479+0000","last_undegraded":"2024-03-23T14:35:14.991479+0000","last_fullsized":"2024-03-23T14:35:14.991479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774293+0000","last_change":"2024-03-23T14:35:13.778143+0000","last_active":"2024-03-23T14:35:14.774293+0000","last_peered":"2024-03-23T14:35:14.774293+0000","last_clean":"2024-03-23T14:35:14.774293+0000","last_became_active":"2024-03-23T14:35:13.776770+0000","last_became_peered":"2024-03-23T14:35:13.776770+0000","last_unstale":"2024-03-23T14:35:14.774293+0000","last_undegraded":"2024-03-23T14:35:14.774293+0000","last_fullsized":"2024-03-23T14:35:14.774293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916781+0000","last_change":"2024-03-23T14:35:12.776956+0000","last_active":"2024-03-23T14:35:14.916781+0000","last_peered":"2024-03-23T14:35:14.916781+0000","last_clean":"2024-03-23T14:35:14.916781+0000","last_became_active":"2024-03-23T14:35:12.776658+0000","last_became_peered":"2024-03-23T14:35:12.776658+0000","last_unstale":"2024-03-23T14:35:14.916781+0000","last_undegraded":"2024-03-23T14:35:14.916781+0000","last_fullsized":"2024-03-23T14:35:14.916781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991420+0000","last_change":"2024-03-23T14:35:13.786795+0000","last_active":"2024-03-23T14:35:14.991420+0000","last_peered":"2024-03-23T14:35:14.991420+0000","last_clean":"2024-03-23T14:35:14.991420+0000","last_became_active":"2024-03-23T14:35:13.786533+0000","last_became_peered":"2024-03-23T14:35:13.786533+0000","last_unstale":"2024-03-23T14:35:14.991420+0000","last_undegraded":"2024-03-23T14:35:14.991420+0000","last_fullsized":"2024-03-23T14:35:14.991420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:19.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991421+0000","last_change":"2024-03-23T14:35:12.779292+0000","last_active":"2024-03-23T14:35:14.991421+0000","last_peered":"2024-03-23T14:35:14.991421+0000","last_clean":"2024-03-23T14:35:14.991421+0000","last_became_active":"2024-03-23T14:35:12.778957+0000","last_became_peered":"2024-03-23T14:35:12.778957+0000","last_unstale":"2024-03-23T14:35:14.991421+0000","last_undegraded":"2024-03-23T14:35:14.991421+0000","last_fullsized":"2024-03-23T14:35:14.991421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991873+0000","last_change":"2024-03-23T14:35:13.785374+0000","last_active":"2024-03-23T14:35:14.991873+0000","last_peered":"2024-03-23T14:35:14.991873+0000","last_clean":"2024-03-23T14:35:14.991873+0000","last_became_active":"2024-03-23T14:35:13.784972+0000","last_became_peered":"2024-03-23T14:35:13.784972+0000","last_unstale":"2024-03-23T14:35:14.991873+0000","last_undegraded":"2024-03-23T14:35:14.991873+0000","last_fullsized":"2024-03-23T14:35:14.991873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774494+0000","last_change":"2024-03-23T14:35:12.775269+0000","last_active":"2024-03-23T14:35:14.774494+0000","last_peered":"2024-03-23T14:35:14.774494+0000","last_clean":"2024-03-23T14:35:14.774494+0000","last_became_active":"2024-03-23T14:35:12.774985+0000","last_became_peered":"2024-03-23T14:35:12.774985+0000","last_unstale":"2024-03-23T14:35:14.774494+0000","last_undegraded":"2024-03-23T14:35:14.774494+0000","last_fullsized":"2024-03-23T14:35:14.774494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775426+0000","last_change":"2024-03-23T14:35:13.777078+0000","last_active":"2024-03-23T14:35:14.775426+0000","last_peered":"2024-03-23T14:35:14.775426+0000","last_clean":"2024-03-23T14:35:14.775426+0000","last_became_active":"2024-03-23T14:35:13.776801+0000","last_became_peered":"2024-03-23T14:35:13.776801+0000","last_unstale":"2024-03-23T14:35:14.775426+0000","last_undegraded":"2024-03-23T14:35:14.775426+0000","last_fullsized":"2024-03-23T14:35:14.775426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:51.829515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916705+0000","last_change":"2024-03-23T14:35:12.773980+0000","last_active":"2024-03-23T14:35:14.916705+0000","last_peered":"2024-03-23T14:35:14.916705+0000","last_clean":"2024-03-23T14:35:14.916705+0000","last_became_active":"2024-03-23T14:35:12.773168+0000","last_became_peered":"2024-03-23T14:35:12.773168+0000","last_unstale":"2024-03-23T14:35:14.916705+0000","last_undegraded":"2024-03-23T14:35:14.916705+0000","last_fullsized":"2024-03-23T14:35:14.916705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991870+0000","last_change":"2024-03-23T14:35:13.789833+0000","last_active":"2024-03-23T14:35:14.991870+0000","last_peered":"2024-03-23T14:35:14.991870+0000","last_clean":"2024-03-23T14:35:14.991870+0000","last_became_active":"2024-03-23T14:35:13.789586+0000","last_became_peered":"2024-03-23T14:35:13.789586+0000","last_unstale":"2024-03-23T14:35:14.991870+0000","last_undegraded":"2024-03-23T14:35:14.991870+0000","last_fullsized":"2024-03-23T14:35:14.991870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991752+0000","last_change":"2024-03-23T14:35:12.773078+0000","last_active":"2024-03-23T14:35:14.991752+0000","last_peered":"2024-03-23T14:35:14.991752+0000","last_clean":"2024-03-23T14:35:14.991752+0000","last_became_active":"2024-03-23T14:35:12.772785+0000","last_became_peered":"2024-03-23T14:35:12.772785+0000","last_unstale":"2024-03-23T14:35:14.991752+0000","last_undegraded":"2024-03-23T14:35:14.991752+0000","last_fullsized":"2024-03-23T14:35:14.991752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:33:10.353085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991346+0000","last_change":"2024-03-23T14:35:13.784377+0000","last_active":"2024-03-23T14:35:14.991346+0000","last_peered":"2024-03-23T14:35:14.991346+0000","last_clean":"2024-03-23T14:35:14.991346+0000","last_became_active":"2024-03-23T14:35:13.783490+0000","last_became_peered":"2024-03-23T14:35:13.783490+0000","last_unstale":"2024-03-23T14:35:14.991346+0000","last_undegraded":"2024-03-23T14:35:14.991346+0000","last_fullsized":"2024-03-23T14:35:14.991346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:37.554855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991229+0000","last_change":"2024-03-23T14:35:12.780946+0000","last_active":"2024-03-23T14:35:14.991229+0000","last_peered":"2024-03-23T14:35:14.991229+0000","last_clean":"2024-03-23T14:35:14.991229+0000","last_became_active":"2024-03-23T14:35:12.780648+0000","last_became_peered":"2024-03-23T14:35:12.780648+0000","last_unstale":"2024-03-23T14:35:14.991229+0000","last_undegraded":"2024-03-23T14:35:14.991229+0000","last_fullsized":"2024-03-23T14:35:14.991229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992120+0000","last_change":"2024-03-23T14:35:13.784062+0000","last_active":"2024-03-23T14:35:14.992120+0000","last_peered":"2024-03-23T14:35:14.992120+0000","last_clean":"2024-03-23T14:35:14.992120+0000","last_became_active":"2024-03-23T14:35:13.783777+0000","last_became_peered":"2024-03-23T14:35:13.783777+0000","last_unstale":"2024-03-23T14:35:14.992120+0000","last_undegraded":"2024-03-23T14:35:14.992120+0000","last_fullsized":"2024-03-23T14:35:14.992120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:31:20.667310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916665+0000","last_change":"2024-03-23T14:35:12.772698+0000","last_active":"2024-03-23T14:35:14.916665+0000","last_peered":"2024-03-23T14:35:14.916665+0000","last_clean":"2024-03-23T14:35:14.916665+0000","last_became_active":"2024-03-23T14:35:12.772263+0000","last_became_peered":"2024-03-23T14:35:12.772263+0000","last_unstale":"2024-03-23T14:35:14.916665+0000","last_undegraded":"2024-03-23T14:35:14.916665+0000","last_fullsized":"2024-03-23T14:35:14.916665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916589+0000","last_change":"2024-03-23T14:35:13.785748+0000","last_active":"2024-03-23T14:35:14.916589+0000","last_peered":"2024-03-23T14:35:14.916589+0000","last_clean":"2024-03-23T14:35:14.916589+0000","last_became_active":"2024-03-23T14:35:13.785468+0000","last_became_peered":"2024-03-23T14:35:13.785468+0000","last_unstale":"2024-03-23T14:35:14.916589+0000","last_undegraded":"2024-03-23T14:35:14.916589+0000","last_fullsized":"2024-03-23T14:35:14.916589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771407+0000","last_change":"2024-03-23T14:35:12.766835+0000","last_active":"2024-03-23T14:35:14.771407+0000","last_peered":"2024-03-23T14:35:14.771407+0000","last_clean":"2024-03-23T14:35:14.771407+0000","last_became_active":"2024-03-23T14:35:12.766558+0000","last_became_peered":"2024-03-23T14:35:12.766558+0000","last_unstale":"2024-03-23T14:35:14.771407+0000","last_undegraded":"2024-03-23T14:35:14.771407+0000","last_fullsized":"2024-03-23T14:35:14.771407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:50.295934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991770+0000","last_change":"2024-03-23T14:35:13.783257+0000","last_active":"2024-03-23T14:35:14.991770+0000","last_peered":"2024-03-23T14:35:14.991770+0000","last_clean":"2024-03-23T14:35:14.991770+0000","last_became_active":"2024-03-23T14:35:13.782963+0000","last_became_peered":"2024-03-23T14:35:13.782963+0000","last_unstale":"2024-03-23T14:35:14.991770+0000","last_undegraded":"2024-03-23T14:35:14.991770+0000","last_fullsized":"2024-03-23T14:35:14.991770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775329+0000","last_change":"2024-03-23T14:35:12.769564+0000","last_active":"2024-03-23T14:35:14.775329+0000","last_peered":"2024-03-23T14:35:14.775329+0000","last_clean":"2024-03-23T14:35:14.775329+0000","last_became_active":"2024-03-23T14:35:12.769153+0000","last_became_peered":"2024-03-23T14:35:12.769153+0000","last_unstale":"2024-03-23T14:35:14.775329+0000","last_undegraded":"2024-03-23T14:35:14.775329+0000","last_fullsized":"2024-03-23T14:35:14.775329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:37:12.470551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991883+0000","last_change":"2024-03-23T14:35:13.783901+0000","last_active":"2024-03-23T14:35:14.991883+0000","last_peered":"2024-03-23T14:35:14.991883+0000","last_clean":"2024-03-23T14:35:14.991883+0000","last_became_active":"2024-03-23T14:35:13.783590+0000","last_became_peered":"2024-03-23T14:35:13.783590+0000","last_unstale":"2024-03-23T14:35:14.991883+0000","last_undegraded":"2024-03-23T14:35:14.991883+0000","last_fullsized":"2024-03-23T14:35:14.991883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771642+0000","last_change":"2024-03-23T14:35:12.767718+0000","last_active":"2024-03-23T14:35:14.771642+0000","last_peered":"2024-03-23T14:35:14.771642+0000","last_clean":"2024-03-23T14:35:14.771642+0000","last_became_active":"2024-03-23T14:35:12.767437+0000","last_became_peered":"2024-03-23T14:35:12.767437+0000","last_unstale":"2024-03-23T14:35:14.771642+0000","last_undegraded":"2024-03-23T14:35:14.771642+0000","last_fullsized":"2024-03-23T14:35:14.771642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992143+0000","last_change":"2024-03-23T14:35:13.785178+0000","last_active":"2024-03-23T14:35:14.992143+0000","last_peered":"2024-03-23T14:35:14.992143+0000","last_clean":"2024-03-23T14:35:14.992143+0000","last_became_active":"2024-03-23T14:35:13.781728+0000","last_became_peered":"2024-03-23T14:35:13.781728+0000","last_unstale":"2024-03-23T14:35:14.992143+0000","last_undegraded":"2024-03-23T14:35:14.992143+0000","last_fullsized":"2024-03-23T14:35:14.992143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:26.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774530+0000","last_change":"2024-03-23T14:35:12.764015+0000","last_active":"2024-03-23T14:35:14.774530+0000","last_peered":"2024-03-23T14:35:14.774530+0000","last_clean":"2024-03-23T14:35:14.774530+0000","last_became_active":"2024-03-23T14:35:12.763735+0000","last_became_peered":"2024-03-23T14:35:12.763735+0000","last_unstale":"2024-03-23T14:35:14.774530+0000","last_undegraded":"2024-03-23T14:35:14.774530+0000","last_fullsized":"2024-03-23T14:35:14.774530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:43.686652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991578+0000","last_change":"2024-03-23T14:35:13.780335+0000","last_active":"2024-03-23T14:35:14.991578+0000","last_peered":"2024-03-23T14:35:14.991578+0000","last_clean":"2024-03-23T14:35:14.991578+0000","last_became_active":"2024-03-23T14:35:13.780078+0000","last_became_peered":"2024-03-23T14:35:13.780078+0000","last_unstale":"2024-03-23T14:35:14.991578+0000","last_undegraded":"2024-03-23T14:35:14.991578+0000","last_fullsized":"2024-03-23T14:35:14.991578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:26.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771754+0000","last_change":"2024-03-23T14:35:12.772701+0000","last_active":"2024-03-23T14:35:14.771754+0000","last_peered":"2024-03-23T14:35:14.771754+0000","last_clean":"2024-03-23T14:35:14.771754+0000","last_became_active":"2024-03-23T14:35:12.772375+0000","last_became_peered":"2024-03-23T14:35:12.772375+0000","last_unstale":"2024-03-23T14:35:14.771754+0000","last_undegraded":"2024-03-23T14:35:14.771754+0000","last_fullsized":"2024-03-23T14:35:14.771754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991996+0000","last_change":"2024-03-23T14:35:13.789100+0000","last_active":"2024-03-23T14:35:14.991996+0000","last_peered":"2024-03-23T14:35:14.991996+0000","last_clean":"2024-03-23T14:35:14.991996+0000","last_became_active":"2024-03-23T14:35:13.788744+0000","last_became_peered":"2024-03-23T14:35:13.788744+0000","last_unstale":"2024-03-23T14:35:14.991996+0000","last_undegraded":"2024-03-23T14:35:14.991996+0000","last_fullsized":"2024-03-23T14:35:14.991996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:54:02.413325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991363+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:35:14.991363+0000","last_peered":"2024-03-23T14:35:14.991363+0000","last_clean":"2024-03-23T14:35:14.991363+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:35:14.991363+0000","last_undegraded":"2024-03-23T14:35:14.991363+0000","last_fullsized":"2024-03-23T14:35:14.991363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991027+0000","last_change":"2024-03-23T14:35:12.772428+0000","last_active":"2024-03-23T14:35:14.991027+0000","last_peered":"2024-03-23T14:35:14.991027+0000","last_clean":"2024-03-23T14:35:14.991027+0000","last_became_active":"2024-03-23T14:35:12.772152+0000","last_became_peered":"2024-03-23T14:35:12.772152+0000","last_unstale":"2024-03-23T14:35:14.991027+0000","last_undegraded":"2024-03-23T14:35:14.991027+0000","last_fullsized":"2024-03-23T14:35:14.991027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992036+0000","last_change":"2024-03-23T14:35:13.779880+0000","last_active":"2024-03-23T14:35:14.992036+0000","last_peered":"2024-03-23T14:35:14.992036+0000","last_clean":"2024-03-23T14:35:14.992036+0000","last_became_active":"2024-03-23T14:35:13.779608+0000","last_became_peered":"2024-03-23T14:35:13.779608+0000","last_unstale":"2024-03-23T14:35:14.992036+0000","last_undegraded":"2024-03-23T14:35:14.992036+0000","last_fullsized":"2024-03-23T14:35:14.992036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991307+0000","last_change":"2024-03-23T14:35:12.781319+0000","last_active":"2024-03-23T14:35:14.991307+0000","last_peered":"2024-03-23T14:35:14.991307+0000","last_clean":"2024-03-23T14:35:14.991307+0000","last_became_active":"2024-03-23T14:35:12.780393+0000","last_became_peered":"2024-03-23T14:35:12.780393+0000","last_unstale":"2024-03-23T14:35:14.991307+0000","last_undegraded":"2024-03-23T14:35:14.991307+0000","last_fullsized":"2024-03-23T14:35:14.991307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991191+0000","last_change":"2024-03-23T14:35:13.783755+0000","last_active":"2024-03-23T14:35:14.991191+0000","last_peered":"2024-03-23T14:35:14.991191+0000","last_clean":"2024-03-23T14:35:14.991191+0000","last_became_active":"2024-03-23T14:35:13.783415+0000","last_became_peered":"2024-03-23T14:35:13.783415+0000","last_unstale":"2024-03-23T14:35:14.991191+0000","last_undegraded":"2024-03-23T14:35:14.991191+0000","last_fullsized":"2024-03-23T14:35:14.991191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:07:11.094535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771472+0000","last_change":"2024-03-23T14:35:12.769935+0000","last_active":"2024-03-23T14:35:14.771472+0000","last_peered":"2024-03-23T14:35:14.771472+0000","last_clean":"2024-03-23T14:35:14.771472+0000","last_became_active":"2024-03-23T14:35:12.769638+0000","last_became_peered":"2024-03-23T14:35:12.769638+0000","last_unstale":"2024-03-23T14:35:14.771472+0000","last_undegraded":"2024-03-23T14:35:14.771472+0000","last_fullsized":"2024-03-23T14:35:14.771472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774313+0000","last_change":"2024-03-23T14:35:13.776295+0000","last_active":"2024-03-23T14:35:14.774313+0000","last_peered":"2024-03-23T14:35:14.774313+0000","last_clean":"2024-03-23T14:35:14.774313+0000","last_became_active":"2024-03-23T14:35:13.775968+0000","last_became_peered":"2024-03-23T14:35:13.775968+0000","last_unstale":"2024-03-23T14:35:14.774313+0000","last_undegraded":"2024-03-23T14:35:14.774313+0000","last_fullsized":"2024-03-23T14:35:14.774313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991315+0000","last_change":"2024-03-23T14:35:12.776276+0000","last_active":"2024-03-23T14:35:14.991315+0000","last_peered":"2024-03-23T14:35:14.991315+0000","last_clean":"2024-03-23T14:35:14.991315+0000","last_became_active":"2024-03-23T14:35:12.775990+0000","last_became_peered":"2024-03-23T14:35:12.775990+0000","last_unstale":"2024-03-23T14:35:14.991315+0000","last_undegraded":"2024-03-23T14:35:14.991315+0000","last_fullsized":"2024-03-23T14:35:14.991315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775094+0000","last_change":"2024-03-23T14:35:13.786554+0000","last_active":"2024-03-23T14:35:14.775094+0000","last_peered":"2024-03-23T14:35:14.775094+0000","last_clean":"2024-03-23T14:35:14.775094+0000","last_became_active":"2024-03-23T14:35:13.786286+0000","last_became_peered":"2024-03-23T14:35:13.786286+0000","last_unstale":"2024-03-23T14:35:14.775094+0000","last_undegraded":"2024-03-23T14:35:14.775094+0000","last_fullsized":"2024-03-23T14:35:14.775094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:40.665771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774420+0000","last_change":"2024-03-23T14:35:12.764502+0000","last_active":"2024-03-23T14:35:14.774420+0000","last_peered":"2024-03-23T14:35:14.774420+0000","last_clean":"2024-03-23T14:35:14.774420+0000","last_became_active":"2024-03-23T14:35:12.764241+0000","last_became_peered":"2024-03-23T14:35:12.764241+0000","last_unstale":"2024-03-23T14:35:14.774420+0000","last_undegraded":"2024-03-23T14:35:14.774420+0000","last_fullsized":"2024-03-23T14:35:14.774420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991506+0000","last_change":"2024-03-23T14:35:13.775872+0000","last_active":"2024-03-23T14:35:14.991506+0000","last_peered":"2024-03-23T14:35:14.991506+0000","last_clean":"2024-03-23T14:35:14.991506+0000","last_became_active":"2024-03-23T14:35:13.775527+0000","last_became_peered":"2024-03-23T14:35:13.775527+0000","last_unstale":"2024-03-23T14:35:14.991506+0000","last_undegraded":"2024-03-23T14:35:14.991506+0000","last_fullsized":"2024-03-23T14:35:14.991506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:17:27.919253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992153+0000","last_change":"2024-03-23T14:35:12.780722+0000","last_active":"2024-03-23T14:35:14.992153+0000","last_peered":"2024-03-23T14:35:14.992153+0000","last_clean":"2024-03-23T14:35:14.992153+0000","last_became_active":"2024-03-23T14:35:12.780349+0000","last_became_peered":"2024-03-23T14:35:12.780349+0000","last_unstale":"2024-03-23T14:35:14.992153+0000","last_undegraded":"2024-03-23T14:35:14.992153+0000","last_fullsized":"2024-03-23T14:35:14.992153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:50.822404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774384+0000","last_change":"2024-03-23T14:35:13.776300+0000","last_active":"2024-03-23T14:35:14.774384+0000","last_peered":"2024-03-23T14:35:14.774384+0000","last_clean":"2024-03-23T14:35:14.774384+0000","last_became_active":"2024-03-23T14:35:13.776007+0000","last_became_peered":"2024-03-23T14:35:13.776007+0000","last_unstale":"2024-03-23T14:35:14.774384+0000","last_undegraded":"2024-03-23T14:35:14.774384+0000","last_fullsized":"2024-03-23T14:35:14.774384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774585+0000","last_change":"2024-03-23T14:35:12.772616+0000","last_active":"2024-03-23T14:35:14.774585+0000","last_peered":"2024-03-23T14:35:14.774585+0000","last_clean":"2024-03-23T14:35:14.774585+0000","last_became_active":"2024-03-23T14:35:12.772237+0000","last_became_peered":"2024-03-23T14:35:12.772237+0000","last_unstale":"2024-03-23T14:35:14.774585+0000","last_undegraded":"2024-03-23T14:35:14.774585+0000","last_fullsized":"2024-03-23T14:35:14.774585+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:50.822404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992233+0000","last_change":"2024-03-23T14:35:13.782070+0000","last_active":"2024-03-23T14:35:14.992233+0000","last_peered":"2024-03-23T14:35:14.992233+0000","last_clean":"2024-03-23T14:35:14.992233+0000","last_became_active":"2024-03-23T14:35:13.781798+0000","last_became_peered":"2024-03-23T14:35:13.781798+0000","last_unstale":"2024-03-23T14:35:14.992233+0000","last_undegraded":"2024-03-23T14:35:14.992233+0000","last_fullsized":"2024-03-23T14:35:14.992233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916782+0000","last_change":"2024-03-23T14:35:12.777518+0000","last_active":"2024-03-23T14:35:14.916782+0000","last_peered":"2024-03-23T14:35:14.916782+0000","last_clean":"2024-03-23T14:35:14.916782+0000","last_became_active":"2024-03-23T14:35:12.777242+0000","last_became_peered":"2024-03-23T14:35:12.777242+0000","last_unstale":"2024-03-23T14:35:14.916782+0000","last_undegraded":"2024-03-23T14:35:14.916782+0000","last_fullsized":"2024-03-23T14:35:14.916782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991622+0000","last_change":"2024-03-23T14:35:13.786237+0000","last_active":"2024-03-23T14:35:14.991622+0000","last_peered":"2024-03-23T14:35:14.991622+0000","last_clean":"2024-03-23T14:35:14.991622+0000","last_became_active":"2024-03-23T14:35:13.785920+0000","last_became_peered":"2024-03-23T14:35:13.785920+0000","last_unstale":"2024-03-23T14:35:14.991622+0000","last_undegraded":"2024-03-23T14:35:14.991622+0000","last_fullsized":"2024-03-23T14:35:14.991622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991213+0000","last_change":"2024-03-23T14:35:12.781097+0000","last_active":"2024-03-23T14:35:14.991213+0000","last_peered":"2024-03-23T14:35:14.991213+0000","last_clean":"2024-03-23T14:35:14.991213+0000","last_became_active":"2024-03-23T14:35:12.780408+0000","last_became_peered":"2024-03-23T14:35:12.780408+0000","last_unstale":"2024-03-23T14:35:14.991213+0000","last_undegraded":"2024-03-23T14:35:14.991213+0000","last_fullsized":"2024-03-23T14:35:14.991213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774978+0000","last_change":"2024-03-23T14:35:13.777707+0000","last_active":"2024-03-23T14:35:14.774978+0000","last_peered":"2024-03-23T14:35:14.774978+0000","last_clean":"2024-03-23T14:35:14.774978+0000","last_became_active":"2024-03-23T14:35:13.777470+0000","last_became_peered":"2024-03-23T14:35:13.777470+0000","last_unstale":"2024-03-23T14:35:14.774978+0000","last_undegraded":"2024-03-23T14:35:14.774978+0000","last_fullsized":"2024-03-23T14:35:14.774978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:39.242232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774400+0000","last_change":"2024-03-23T14:35:12.771937+0000","last_active":"2024-03-23T14:35:14.774400+0000","last_peered":"2024-03-23T14:35:14.774400+0000","last_clean":"2024-03-23T14:35:14.774400+0000","last_became_active":"2024-03-23T14:35:12.771560+0000","last_became_peered":"2024-03-23T14:35:12.771560+0000","last_unstale":"2024-03-23T14:35:14.774400+0000","last_undegraded":"2024-03-23T14:35:14.774400+0000","last_fullsized":"2024-03-23T14:35:14.774400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991517+0000","last_change":"2024-03-23T14:35:13.779913+0000","last_active":"2024-03-23T14:35:14.991517+0000","last_peered":"2024-03-23T14:35:14.991517+0000","last_clean":"2024-03-23T14:35:14.991517+0000","last_became_active":"2024-03-23T14:35:13.779667+0000","last_became_peered":"2024-03-23T14:35:13.779667+0000","last_unstale":"2024-03-23T14:35:14.991517+0000","last_undegraded":"2024-03-23T14:35:14.991517+0000","last_fullsized":"2024-03-23T14:35:14.991517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774464+0000","last_change":"2024-03-23T14:35:12.770228+0000","last_active":"2024-03-23T14:35:14.774464+0000","last_peered":"2024-03-23T14:35:14.774464+0000","last_clean":"2024-03-23T14:35:14.774464+0000","last_became_active":"2024-03-23T14:35:12.769962+0000","last_became_peered":"2024-03-23T14:35:12.769962+0000","last_unstale":"2024-03-23T14:35:14.774464+0000","last_undegraded":"2024-03-23T14:35:14.774464+0000","last_fullsized":"2024-03-23T14:35:14.774464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991918+0000","last_change":"2024-03-23T14:35:13.786710+0000","last_active":"2024-03-23T14:35:14.991918+0000","last_peered":"2024-03-23T14:35:14.991918+0000","last_clean":"2024-03-23T14:35:14.991918+0000","last_became_active":"2024-03-23T14:35:13.786454+0000","last_became_peered":"2024-03-23T14:35:13.786454+0000","last_unstale":"2024-03-23T14:35:14.991918+0000","last_undegraded":"2024-03-23T14:35:14.991918+0000","last_fullsized":"2024-03-23T14:35:14.991918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992224+0000","last_change":"2024-03-23T14:35:12.778699+0000","last_active":"2024-03-23T14:35:14.992224+0000","last_peered":"2024-03-23T14:35:14.992224+0000","last_clean":"2024-03-23T14:35:14.992224+0000","last_became_active":"2024-03-23T14:35:12.778416+0000","last_became_peered":"2024-03-23T14:35:12.778416+0000","last_unstale":"2024-03-23T14:35:14.992224+0000","last_undegraded":"2024-03-23T14:35:14.992224+0000","last_fullsized":"2024-03-23T14:35:14.992224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916656+0000","last_change":"2024-03-23T14:35:13.773983+0000","last_active":"2024-03-23T14:35:14.916656+0000","last_peered":"2024-03-23T14:35:14.916656+0000","last_clean":"2024-03-23T14:35:14.916656+0000","last_became_active":"2024-03-23T14:35:13.773689+0000","last_became_peered":"2024-03-23T14:35:13.773689+0000","last_unstale":"2024-03-23T14:35:14.916656+0000","last_undegraded":"2024-03-23T14:35:14.916656+0000","last_fullsized":"2024-03-23T14:35:14.916656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991352+0000","last_change":"2024-03-23T14:35:13.783572+0000","last_active":"2024-03-23T14:35:14.991352+0000","last_peered":"2024-03-23T14:35:14.991352+0000","last_clean":"2024-03-23T14:35:14.991352+0000","last_became_active":"2024-03-23T14:35:13.782224+0000","last_became_peered":"2024-03-23T14:35:13.782224+0000","last_unstale":"2024-03-23T14:35:14.991352+0000","last_undegraded":"2024-03-23T14:35:14.991352+0000","last_fullsized":"2024-03-23T14:35:14.991352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:20.569093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991763+0000","last_change":"2024-03-23T14:35:12.775487+0000","last_active":"2024-03-23T14:35:14.991763+0000","last_peered":"2024-03-23T14:35:14.991763+0000","last_clean":"2024-03-23T14:35:14.991763+0000","last_became_active":"2024-03-23T14:35:12.775208+0000","last_became_peered":"2024-03-23T14:35:12.775208+0000","last_unstale":"2024-03-23T14:35:14.991763+0000","last_undegraded":"2024-03-23T14:35:14.991763+0000","last_fullsized":"2024-03-23T14:35:14.991763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991431+0000","last_change":"2024-03-23T14:35:13.783879+0000","last_active":"2024-03-23T14:35:14.991431+0000","last_peered":"2024-03-23T14:35:14.991431+0000","last_clean":"2024-03-23T14:35:14.991431+0000","last_became_active":"2024-03-23T14:35:13.783451+0000","last_became_peered":"2024-03-23T14:35:13.783451+0000","last_unstale":"2024-03-23T14:35:14.991431+0000","last_undegraded":"2024-03-23T14:35:14.991431+0000","last_fullsized":"2024-03-23T14:35:14.991431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:54:02.413325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992060+0000","last_change":"2024-03-23T14:35:12.777520+0000","last_active":"2024-03-23T14:35:14.992060+0000","last_peered":"2024-03-23T14:35:14.992060+0000","last_clean":"2024-03-23T14:35:14.992060+0000","last_became_active":"2024-03-23T14:35:12.777254+0000","last_became_peered":"2024-03-23T14:35:12.777254+0000","last_unstale":"2024-03-23T14:35:14.992060+0000","last_undegraded":"2024-03-23T14:35:14.992060+0000","last_fullsized":"2024-03-23T14:35:14.992060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:43.686652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991341+0000","last_change":"2024-03-23T14:35:13.785243+0000","last_active":"2024-03-23T14:35:14.991341+0000","last_peered":"2024-03-23T14:35:14.991341+0000","last_clean":"2024-03-23T14:35:14.991341+0000","last_became_active":"2024-03-23T14:35:13.784972+0000","last_became_peered":"2024-03-23T14:35:13.784972+0000","last_unstale":"2024-03-23T14:35:14.991341+0000","last_undegraded":"2024-03-23T14:35:14.991341+0000","last_fullsized":"2024-03-23T14:35:14.991341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:24:06.436157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992114+0000","last_change":"2024-03-23T14:35:12.770618+0000","last_active":"2024-03-23T14:35:14.992114+0000","last_peered":"2024-03-23T14:35:14.992114+0000","last_clean":"2024-03-23T14:35:14.992114+0000","last_became_active":"2024-03-23T14:35:12.770320+0000","last_became_peered":"2024-03-23T14:35:12.770320+0000","last_unstale":"2024-03-23T14:35:14.992114+0000","last_undegraded":"2024-03-23T14:35:14.992114+0000","last_fullsized":"2024-03-23T14:35:14.992114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:33:10.353085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991383+0000","last_change":"2024-03-23T14:35:13.775981+0000","last_active":"2024-03-23T14:35:14.991383+0000","last_peered":"2024-03-23T14:35:14.991383+0000","last_clean":"2024-03-23T14:35:14.991383+0000","last_became_active":"2024-03-23T14:35:13.775695+0000","last_became_peered":"2024-03-23T14:35:13.775695+0000","last_unstale":"2024-03-23T14:35:14.991383+0000","last_undegraded":"2024-03-23T14:35:14.991383+0000","last_fullsized":"2024-03-23T14:35:14.991383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991901+0000","last_change":"2024-03-23T14:35:12.773386+0000","last_active":"2024-03-23T14:35:14.991901+0000","last_peered":"2024-03-23T14:35:14.991901+0000","last_clean":"2024-03-23T14:35:14.991901+0000","last_became_active":"2024-03-23T14:35:12.773104+0000","last_became_peered":"2024-03-23T14:35:12.773104+0000","last_unstale":"2024-03-23T14:35:14.991901+0000","last_undegraded":"2024-03-23T14:35:14.991901+0000","last_fullsized":"2024-03-23T14:35:14.991901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:34.363745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574854,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574853,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27216,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344624,"statfs":{"total":96636764160,"available":96608894976,"internally_reserved":0,"allocated":196608,"data_stored":64724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574853,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574855,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":196,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":200704,"data_stored":64794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574854,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:35:17.072 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T14:35:17.072 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-23T14:35:17.072 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-03-23T14:35:17.072 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-03-23T14:35:17.073 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-03-23T14:35:17.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.213+0000 7fa4a19af640 1 Processor -- start 2024-03-23T14:35:17.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.213+0000 7fa4a19af640 1 -- start start 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa4a19af640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa49c15e3b0 0x7fa49c0d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa4a19af640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa49c0b34f0 con 0x7fa49c15e790 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa4a19af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa49c0d4720 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa4a19af640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa49c0d0420 con 0x7fa49c15e3b0 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa49affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa49c15e3b0 0x7fa49c0d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa49a7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa49c15e790 0x7fa49c0b39d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49834/0 (socket says 172.21.15.161:49834) 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa49a7fc640 1 -- 172.21.15.161:0/2326469467 learned_addr learned my addr 172.21.15.161:0/2326469467 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa49affd640 1 -- 172.21.15.161:0/2326469467 >> v1:172.21.15.183:6789/0 conn(0x7fa49c0b2ee0 legacy=0x7fa49c0cfb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa49affd640 1 -- 172.21.15.161:0/2326469467 >> v1:172.21.15.161:6789/0 conn(0x7fa49c15e790 legacy=0x7fa49c0b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa499ffb640 1 -- 172.21.15.161:0/2326469467 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2246811790 0 0) 0x7fa49c0d4720 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.214+0000 7fa49affd640 1 -- 172.21.15.161:0/2326469467 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa49c0d4a00 con 0x7fa49c15e3b0 2024-03-23T14:35:17.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.215+0000 7fa49affd640 1 --2- 172.21.15.161:0/2326469467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c15e3b0 0x7fa49c0d2460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0097a0 tx=0x7fa48c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93003eae9e27cc45 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.215+0000 7fa499ffb640 1 -- 172.21.15.161:0/2326469467 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c030710 con 0x7fa49c15e3b0 2024-03-23T14:35:17.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.215+0000 7fa499ffb640 1 -- 172.21.15.161:0/2326469467 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48c02e050 con 0x7fa49c15e3b0 2024-03-23T14:35:17.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.215+0000 7fa499ffb640 1 -- 172.21.15.161:0/2326469467 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c030a90 con 0x7fa49c15e3b0 2024-03-23T14:35:17.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.216+0000 7fa4a19af640 1 -- 172.21.15.161:0/2326469467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c15e3b0 msgr2=0x7fa49c0d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.216+0000 7fa4a19af640 1 --2- 172.21.15.161:0/2326469467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c15e3b0 0x7fa49c0d2460 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0097a0 tx=0x7fa48c02b790 comp rx=0 tx=0).stop 2024-03-23T14:35:17.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.216+0000 7fa4a19af640 1 -- 172.21.15.161:0/2326469467 shutdown_connections 2024-03-23T14:35:17.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.216+0000 7fa4a19af640 1 --2- 172.21.15.161:0/2326469467 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c15e3b0 0x7fa49c0d2460 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.216+0000 7fa4a19af640 1 -- 172.21.15.161:0/2326469467 >> 172.21.15.161:0/2326469467 conn(0x7fa49c09d1a0 msgr2=0x7fa49c0d3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:17.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.216+0000 7fa4a19af640 1 -- 172.21.15.161:0/2326469467 shutdown_connections 2024-03-23T14:35:17.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.216+0000 7fa4a19af640 1 -- 172.21.15.161:0/2326469467 wait complete. 2024-03-23T14:35:17.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.217+0000 7fa4a19af640 1 Processor -- start 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.217+0000 7fa4a19af640 1 -- start start 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa4a19af640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c0b2ee0 0x7fa49c1522f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa4a19af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa49c15e3b0 0x7fa49c0b8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa4a19af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa49c15e790 0x7fa49c15fef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa4a19af640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c15eb70 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa4a19af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa49c15ee50 con 0x7fa49c15e3b0 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa4a19af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa49c15f130 con 0x7fa49c15e790 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c0b2ee0 0x7fa49c1522f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c0b2ee0 0x7fa49c1522f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41040/0 (socket says 172.21.15.161:41040) 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 -- 172.21.15.161:0/3603974883 learned_addr learned my addr 172.21.15.161:0/3603974883 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:17.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49a7fc640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa49c15e3b0 0x7fa49c0b8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49b7fe640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa49c15e790 0x7fa49c15fef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 -- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa49c15e790 msgr2=0x7fa49c15fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa49c15e790 0x7fa49c15fef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 -- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa49c15e3b0 msgr2=0x7fa49c0b8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa49c15e3b0 0x7fa49c0b8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 -- 172.21.15.161:0/3603974883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa48c009400 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa49affd640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c0b2ee0 0x7fa49c1522f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0097a0 tx=0x7fa48c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa483fff640 1 -- 172.21.15.161:0/3603974883 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c004230 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa483fff640 1 -- 172.21.15.161:0/3603974883 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48c0043b0 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa49c15f410 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa483fff640 1 -- 172.21.15.161:0/3603974883 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c004520 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.218+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa49c15f9d0 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.219+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa49c0a1e80 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.220+0000 7fa483fff640 1 -- 172.21.15.161:0/3603974883 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa48c039040 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.220+0000 7fa483fff640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa48806b810 0x7fa48806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.221+0000 7fa483fff640 1 -- 172.21.15.161:0/3603974883 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa48c0ac030 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.221+0000 7fa49a7fc640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa48806b810 0x7fa48806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.221+0000 7fa49a7fc640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa48806b810 0x7fa48806dcd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0b95b0 tx=0x7fa490007e20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.225+0000 7fa483fff640 1 -- 172.21.15.161:0/3603974883 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa48c076b00 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.446+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fa49c15fbc0 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.448+0000 7fa483fff640 1 -- 172.21.15.161:0/3603974883 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+171791 (secure 0 0 0) 0x7fa48c037090 con 0x7fa49c0b2ee0 2024-03-23T14:35:17.446 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:17.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa48806b810 msgr2=0x7fa48806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa48806b810 0x7fa48806dcd0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa49c0b95b0 tx=0x7fa490007e20 comp rx=0 tx=0).stop 2024-03-23T14:35:17.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c0b2ee0 msgr2=0x7fa49c1522f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c0b2ee0 0x7fa49c1522f0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa48c0097a0 tx=0x7fa48c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 shutdown_connections 2024-03-23T14:35:17.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa48806b810 0x7fa48806dcd0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa49c15e790 0x7fa49c15fef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa49c15e3b0 0x7fa49c0b8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 --2- 172.21.15.161:0/3603974883 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa49c0b2ee0 0x7fa49c1522f0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 >> 172.21.15.161:0/3603974883 conn(0x7fa49c09d1a0 msgr2=0x7fa49c0ceb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:17.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 shutdown_connections 2024-03-23T14:35:17.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.451+0000 7fa4a19af640 1 -- 172.21.15.161:0/3603974883 wait complete. 2024-03-23T14:35:17.458 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6832","nonce":3827122778},{"type":"v1","addr":"172.21.15.183:6833","nonce":3827122778}]},"active_addr":"172.21.15.183:6833/3827122778","active_change":"2024-03-23T14:34:52.465954+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4105,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:latest","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3077665299}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":2233446785}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":3430180617}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.183:0","nonce":492196124}]}]} 2024-03-23T14:35:17.461 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-03-23T14:35:17.461 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-03-23T14:35:17.461 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:17.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.566+0000 7fa62893e640 1 Processor -- start 2024-03-23T14:35:17.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.566+0000 7fa62893e640 1 -- start start 2024-03-23T14:35:17.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.566+0000 7fa62893e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6240724e0 0x7fa62406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.566+0000 7fa62893e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62415be70 con 0x7fa624072100 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa62893e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa62415c150 con 0x7fa62406ead0 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa62893e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa62406ef80 con 0x7fa6240724e0 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa62259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa624072100 0x7fa62415b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49850/0 (socket says 172.21.15.161:49850) 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa62259c640 1 -- 172.21.15.161:0/83463325 learned_addr learned my addr 172.21.15.161:0/83463325 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa619d9b640 1 --2- 172.21.15.161:0/83463325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6240724e0 0x7fa62406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa621d9b640 1 -- 172.21.15.161:0/83463325 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2701436879 0 0) 0x7fa62415be70 con 0x7fa624072100 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa621d9b640 1 -- 172.21.15.161:0/83463325 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa610003520 con 0x7fa624072100 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa621d9b640 1 -- 172.21.15.161:0/83463325 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 329362534 0 0) 0x7fa62415c150 con 0x7fa62406ead0 2024-03-23T14:35:17.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa621d9b640 1 -- 172.21.15.161:0/83463325 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa62415be70 con 0x7fa62406ead0 2024-03-23T14:35:17.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa619d9b640 1 -- 172.21.15.161:0/83463325 >> v1:172.21.15.183:6789/0 conn(0x7fa62406ead0 legacy=0x7fa6240b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa619d9b640 1 -- 172.21.15.161:0/83463325 >> v1:172.21.15.161:6789/0 conn(0x7fa624072100 legacy=0x7fa62415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.567+0000 7fa619d9b640 1 -- 172.21.15.161:0/83463325 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6240b2c70 con 0x7fa6240724e0 2024-03-23T14:35:17.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa619d9b640 1 --2- 172.21.15.161:0/83463325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6240724e0 0x7fa62406e590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa60c0097d0 tx=0x7fa60c02b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2e3a2dbd226be58 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa621d9b640 1 -- 172.21.15.161:0/83463325 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c031890 con 0x7fa6240724e0 2024-03-23T14:35:17.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa621d9b640 1 -- 172.21.15.161:0/83463325 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa60c0319f0 con 0x7fa6240724e0 2024-03-23T14:35:17.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa621d9b640 1 -- 172.21.15.161:0/83463325 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60c031d80 con 0x7fa6240724e0 2024-03-23T14:35:17.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa62893e640 1 -- 172.21.15.161:0/83463325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6240724e0 msgr2=0x7fa62406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa62893e640 1 --2- 172.21.15.161:0/83463325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6240724e0 0x7fa62406e590 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa60c0097d0 tx=0x7fa60c02b930 comp rx=0 tx=0).stop 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa62893e640 1 -- 172.21.15.161:0/83463325 shutdown_connections 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa62893e640 1 --2- 172.21.15.161:0/83463325 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6240724e0 0x7fa62406e590 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa62893e640 1 -- 172.21.15.161:0/83463325 >> 172.21.15.161:0/83463325 conn(0x7fa6240b17f0 msgr2=0x7fa6240b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.568+0000 7fa62893e640 1 -- 172.21.15.161:0/83463325 shutdown_connections 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.569+0000 7fa62893e640 1 -- 172.21.15.161:0/83463325 wait complete. 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.570+0000 7fa62893e640 1 Processor -- start 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.570+0000 7fa62893e640 1 -- start start 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62893e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62406ead0 0x7fa624148e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62893e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa624072100 0x7fa624152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62893e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6240724e0 0x7fa6241f52b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62893e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa62414a280 con 0x7fa62406ead0 2024-03-23T14:35:17.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62893e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa6241493a0 con 0x7fa6240724e0 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62893e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa624149670 con 0x7fa624072100 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62406ead0 0x7fa624148e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62406ead0 0x7fa624148e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41046/0 (socket says 172.21.15.161:41046) 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 -- 172.21.15.161:0/2146666018 learned_addr learned my addr 172.21.15.161:0/2146666018 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa622d9d640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6240724e0 0x7fa6241f52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa619d9b640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa624072100 0x7fa624152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 -- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa624072100 msgr2=0x7fa624152aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa624072100 0x7fa624152aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 -- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6240724e0 msgr2=0x7fa6241f52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6240724e0 0x7fa6241f52b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 -- 172.21.15.161:0/2146666018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa60c009400 con 0x7fa62406ead0 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62259c640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62406ead0 0x7fa624148e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c500 tx=0x7fa61400c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa61b7fe640 1 -- 172.21.15.161:0/2146666018 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa614005ce0 con 0x7fa62406ead0 2024-03-23T14:35:17.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa624149950 con 0x7fa62406ead0 2024-03-23T14:35:17.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa61b7fe640 1 -- 172.21.15.161:0/2146666018 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa614005e40 con 0x7fa62406ead0 2024-03-23T14:35:17.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.571+0000 7fa61b7fe640 1 -- 172.21.15.161:0/2146666018 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6140072f0 con 0x7fa62406ead0 2024-03-23T14:35:17.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.572+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa624149f10 con 0x7fa62406ead0 2024-03-23T14:35:17.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.572+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6240735e0 con 0x7fa62406ead0 2024-03-23T14:35:17.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.573+0000 7fa61b7fe640 1 -- 172.21.15.161:0/2146666018 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa614007450 con 0x7fa62406ead0 2024-03-23T14:35:17.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.573+0000 7fa61b7fe640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa60406b810 0x7fa60406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.573+0000 7fa61b7fe640 1 -- 172.21.15.161:0/2146666018 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa614089790 con 0x7fa62406ead0 2024-03-23T14:35:17.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.573+0000 7fa619d9b640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa60406b810 0x7fa60406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.574+0000 7fa619d9b640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa60406b810 0x7fa60406dcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa60c036000 tx=0x7fa60c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.578+0000 7fa61b7fe640 1 -- 172.21.15.161:0/2146666018 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa614054270 con 0x7fa62406ead0 2024-03-23T14:35:17.731 INFO:tasks.ceph.mds.b.smithi183.stderr:2024-03-23T14:35:17.735+0000 7f9a0490f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T14:35:17.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.750+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa6240cbf90 con 0x7fa62406ead0 2024-03-23T14:35:17.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.752+0000 7fa61b7fe640 1 -- 172.21.15.161:0/2146666018 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7fa614057930 con 0x7fa62406ead0 2024-03-23T14:35:17.748 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:17.748 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:17.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa60406b810 msgr2=0x7fa60406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa60406b810 0x7fa60406dcd0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fa60c036000 tx=0x7fa60c03c040 comp rx=0 tx=0).stop 2024-03-23T14:35:17.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62406ead0 msgr2=0x7fa624148e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62406ead0 0x7fa624148e60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa61400c500 tx=0x7fa61400c9c0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 shutdown_connections 2024-03-23T14:35:17.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa60406b810 0x7fa60406dcd0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6240724e0 0x7fa6241f52b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa624072100 0x7fa624152aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 --2- 172.21.15.161:0/2146666018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa62406ead0 0x7fa624148e60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 >> 172.21.15.161:0/2146666018 conn(0x7fa6240b17f0 msgr2=0x7fa6240cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:17.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 shutdown_connections 2024-03-23T14:35:17.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.754+0000 7fa62893e640 1 -- 172.21.15.161:0/2146666018 wait complete. 2024-03-23T14:35:17.763 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-03-23T14:35:17.763 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.868+0000 7fc2b7fd0640 1 Processor -- start 2024-03-23T14:35:17.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.868+0000 7fc2b7fd0640 1 -- start start 2024-03-23T14:35:17.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b7fd0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b0145080 0x7fc2b0072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b7fd0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc2b01607d0 0x7fc2b015fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b7fd0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2b00cf4f0 con 0x7fc2b006cbf0 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b7fd0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc2b0071010 con 0x7fc2b01607d0 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b7fd0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b00cef30 con 0x7fc2b0145080 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b556b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc2b01607d0 0x7fc2b015fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b556b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc2b01607d0 0x7fc2b015fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41050/0 (socket says 172.21.15.161:41050) 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b656d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc2b006cbf0 0x7fc2b0071430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49852/0 (socket says 172.21.15.161:49852) 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b556b640 1 -- 172.21.15.161:0/2235992082 learned_addr learned my addr 172.21.15.161:0/2235992082 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b5d6c640 1 --2- 172.21.15.161:0/2235992082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b0145080 0x7fc2b0072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b556b640 1 -- 172.21.15.161:0/2235992082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b0145080 msgr2=0x7fc2b0072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b556b640 1 --2- 172.21.15.161:0/2235992082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b0145080 0x7fc2b0072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b556b640 1 -- 172.21.15.161:0/2235992082 >> v1:172.21.15.161:6789/0 conn(0x7fc2b006cbf0 legacy=0x7fc2b0071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b4d6a640 1 -- 172.21.15.161:0/2235992082 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2410565683 0 0) 0x7fc2b00cf4f0 con 0x7fc2b006cbf0 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.869+0000 7fc2b556b640 1 -- 172.21.15.161:0/2235992082 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b00cf210 con 0x7fc2b01607d0 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.870+0000 7fc2b556b640 1 --2- 172.21.15.161:0/2235992082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b01607d0 0x7fc2b015fae0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009870 tx=0x7fc2a002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5035511c77ec609c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.870+0000 7fc2b4d6a640 1 -- 172.21.15.161:0/2235992082 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0031640 con 0x7fc2b01607d0 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.870+0000 7fc2b4d6a640 1 -- 172.21.15.161:0/2235992082 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2a00317a0 con 0x7fc2b01607d0 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.870+0000 7fc2b4d6a640 1 -- 172.21.15.161:0/2235992082 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2a0031b30 con 0x7fc2b01607d0 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.871+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2235992082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b01607d0 msgr2=0x7fc2b015fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.871+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2235992082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b01607d0 0x7fc2b015fae0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0009870 tx=0x7fc2a002b8f0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.871+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2235992082 shutdown_connections 2024-03-23T14:35:17.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.871+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2235992082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b01607d0 0x7fc2b015fae0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.871+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2235992082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b0145080 0x7fc2b0072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.871+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2235992082 >> 172.21.15.161:0/2235992082 conn(0x7fc2b00b17f0 msgr2=0x7fc2b00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:17.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.871+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2235992082 shutdown_connections 2024-03-23T14:35:17.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.871+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2235992082 wait complete. 2024-03-23T14:35:17.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.872+0000 7fc2b7fd0640 1 Processor -- start 2024-03-23T14:35:17.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.872+0000 7fc2b7fd0640 1 -- start start 2024-03-23T14:35:17.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b7fd0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b006cbf0 0x7fc2b01f28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b7fd0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0145080 0x7fc2b01f4670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b7fd0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b01607d0 0x7fc2b01f1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b7fd0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01402b0 con 0x7fc2b006cbf0 2024-03-23T14:35:17.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b7fd0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b0140500 con 0x7fc2b0145080 2024-03-23T14:35:17.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b7fd0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc2b01407e0 con 0x7fc2b01607d0 2024-03-23T14:35:17.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b006cbf0 0x7fc2b01f28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b006cbf0 0x7fc2b01f28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41058/0 (socket says 172.21.15.161:41058) 2024-03-23T14:35:17.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 -- 172.21.15.161:0/2697058868 learned_addr learned my addr 172.21.15.161:0/2697058868 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b656d640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b01607d0 0x7fc2b01f1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b556b640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0145080 0x7fc2b01f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 -- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b01607d0 msgr2=0x7fc2b01f1e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b01607d0 0x7fc2b01f1e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 -- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0145080 msgr2=0x7fc2b01f4670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:17.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0145080 0x7fc2b01f4670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:17.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 -- 172.21.15.161:0/2697058868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2a00094d0 con 0x7fc2b006cbf0 2024-03-23T14:35:17.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b5d6c640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b006cbf0 0x7fc2b01f28a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00e9b0 tx=0x7fc2ac00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2a67fc640 1 -- 172.21.15.161:0/2697058868 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac009670 con 0x7fc2b006cbf0 2024-03-23T14:35:17.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2a67fc640 1 -- 172.21.15.161:0/2697058868 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc2ac00c9a0 con 0x7fc2b006cbf0 2024-03-23T14:35:17.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b0140ac0 con 0x7fc2b006cbf0 2024-03-23T14:35:17.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2a67fc640 1 -- 172.21.15.161:0/2697058868 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc2ac00cd50 con 0x7fc2b006cbf0 2024-03-23T14:35:17.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.873+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b01f23b0 con 0x7fc2b006cbf0 2024-03-23T14:35:17.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.874+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b00735e0 con 0x7fc2b006cbf0 2024-03-23T14:35:17.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.875+0000 7fc2a67fc640 1 -- 172.21.15.161:0/2697058868 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc2ac013470 con 0x7fc2b006cbf0 2024-03-23T14:35:17.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.875+0000 7fc2a67fc640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc29406b810 0x7fc29406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:17.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.875+0000 7fc2a67fc640 1 -- 172.21.15.161:0/2697058868 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc2ac08a400 con 0x7fc2b006cbf0 2024-03-23T14:35:17.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.875+0000 7fc2b556b640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc29406b810 0x7fc29406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:17.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.876+0000 7fc2b556b640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc29406b810 0x7fc29406dcd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0036000 tx=0x7fc2a003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:17.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:17.879+0000 7fc2a67fc640 1 -- 172.21.15.161:0/2697058868 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc2ac054ed0 con 0x7fc2b006cbf0 2024-03-23T14:35:18.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.054+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc2b004f550 con 0x7fc2b006cbf0 2024-03-23T14:35:18.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.055+0000 7fc2a67fc640 1 -- 172.21.15.161:0/2697058868 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7fc2ac058590 con 0x7fc2b006cbf0 2024-03-23T14:35:18.051 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:18.051 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:18.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc29406b810 msgr2=0x7fc29406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc29406b810 0x7fc29406dcd0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fc2a0036000 tx=0x7fc2a003c040 comp rx=0 tx=0).stop 2024-03-23T14:35:18.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b006cbf0 msgr2=0x7fc2b01f28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b006cbf0 0x7fc2b01f28a0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc2ac00e9b0 tx=0x7fc2ac00ee70 comp rx=0 tx=0).stop 2024-03-23T14:35:18.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 shutdown_connections 2024-03-23T14:35:18.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc29406b810 0x7fc29406dcd0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc2b01607d0 0x7fc2b01f1e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc2b0145080 0x7fc2b01f4670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 --2- 172.21.15.161:0/2697058868 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc2b006cbf0 0x7fc2b01f28a0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 >> 172.21.15.161:0/2697058868 conn(0x7fc2b00b17f0 msgr2=0x7fc2b00b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 shutdown_connections 2024-03-23T14:35:18.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.057+0000 7fc2b7fd0640 1 -- 172.21.15.161:0/2697058868 wait complete. 2024-03-23T14:35:18.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-03-23T14:35:18.065 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-03-23T14:35:18.066 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-03-23T14:35:18.067 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-03-23T14:35:18.067 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-03-23T14:35:18.068 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-03-23T14:35:18.068 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-03-23T14:35:18.069 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-03-23T14:35:18.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.242+0000 7f11e0c15640 1 Processor -- start 2024-03-23T14:35:18.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.242+0000 7f11e0c15640 1 -- start start 2024-03-23T14:35:18.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7ff14a888640 1 Processor -- start 2024-03-23T14:35:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7ff14a888640 1 -- start start 2024-03-23T14:35:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7f11e0c15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc071360 0x7f11dc06ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7f11e0c15640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f11dc06d340 0x7f11dc0b3160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7f11e0c15640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11dc0cf860 con 0x7f11dc0cf480 2024-03-23T14:35:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7f11e0c15640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f11dc06d7f0 con 0x7f11dc06d340 2024-03-23T14:35:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7f11e0c15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f11dc06f520 con 0x7f11dc071360 2024-03-23T14:35:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7f11da59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f11dc0cf480 0x7f11dc163590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49856/0 (socket says 172.21.15.161:49856) 2024-03-23T14:35:18.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7f11da59c640 1 -- 172.21.15.161:0/2622242242 learned_addr learned my addr 172.21.15.161:0/2622242242 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7fa65642e640 1 Processor -- start 2024-03-23T14:35:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.243+0000 7fa65642e640 1 -- start start 2024-03-23T14:35:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7f11dad9d640 1 --2- 172.21.15.161:0/2622242242 >> v2:172.21.15.183:3300/0 conn(0x7f11dc06d340 0x7f11dc0b3160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7f11d9d9b640 1 --2- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc071360 0x7f11dc06ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7fa65642e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa6500cf480 0x7fa650163590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7fa65642e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa650071360 0x7fa65006ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7fa65642e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa6500cf860 con 0x7fa65006d340 2024-03-23T14:35:18.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7fa65642e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa65006f520 con 0x7fa650071360 2024-03-23T14:35:18.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.244+0000 7fa65642e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa650072470 con 0x7fa6500cf480 2024-03-23T14:35:18.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7f11d959a640 1 -- 172.21.15.161:0/2622242242 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 755580212 0 0) 0x7f11dc0cf860 con 0x7f11dc0cf480 2024-03-23T14:35:18.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff14a888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c006f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff14a888640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff13c007460 0x7ff13c053e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff14a888640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13c0547e0 con 0x7ff13c054390 2024-03-23T14:35:18.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff14a888640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff13c0484e0 con 0x7ff13c007460 2024-03-23T14:35:18.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff14a888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c0078c0 con 0x7ff13c048100 2024-03-23T14:35:18.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7f11d959a640 1 -- 172.21.15.161:0/2622242242 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11c4003520 con 0x7f11dc0cf480 2024-03-23T14:35:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff148e25640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff13c054390 0x7ff13c04f930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33850/0 (socket says 172.21.15.161:33850) 2024-03-23T14:35:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff148e25640 1 -- 172.21.15.161:0/1517147435 learned_addr learned my addr 172.21.15.161:0/1517147435 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff143fff640 1 --2- 172.21.15.161:0/1517147435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c006f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7f11dad9d640 1 -- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc071360 msgr2=0x7f11dc06ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7f11dad9d640 1 --2- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc071360 0x7f11dc06ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff143fff640 1 -- 172.21.15.161:0/1517147435 >> v2:172.21.15.161:3300/0 conn(0x7ff13c007460 msgr2=0x7ff13c053e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff143fff640 1 --2- 172.21.15.161:0/1517147435 >> v2:172.21.15.161:3300/0 conn(0x7ff13c007460 0x7ff13c053e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff143fff640 1 -- 172.21.15.161:0/1517147435 >> v1:172.21.15.183:6789/0 conn(0x7ff13c054390 legacy=0x7ff13c04f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7ff143fff640 1 -- 172.21.15.161:0/1517147435 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13c006a90 con 0x7ff13c048100 2024-03-23T14:35:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7f11dad9d640 1 -- 172.21.15.161:0/2622242242 >> v1:172.21.15.161:6789/0 conn(0x7f11dc0cf480 legacy=0x7f11dc163590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7f11dad9d640 1 -- 172.21.15.161:0/2622242242 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11dc072470 con 0x7f11dc06d340 2024-03-23T14:35:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7f11d9d9b640 1 --2- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc071360 0x7f11dc06ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:18.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fa5fffc7640 1 Processor -- start 2024-03-23T14:35:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fa5fffc7640 1 -- start start 2024-03-23T14:35:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.245+0000 7f18c7c76640 1 Processor -- start 2024-03-23T14:35:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fa6549cb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa65006d340 0x7fa650166100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49858/0 (socket says 172.21.15.161:49858) 2024-03-23T14:35:18.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fa6549cb640 1 -- 172.21.15.161:0/3207661963 learned_addr learned my addr 172.21.15.161:0/3207661963 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fb649009640 1 Processor -- start 2024-03-23T14:35:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fc726de9640 1 Processor -- start 2024-03-23T14:35:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fc726de9640 1 -- start start 2024-03-23T14:35:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fe6de9e2640 1 Processor -- start 2024-03-23T14:35:18.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fe6de9e2640 1 -- start start 2024-03-23T14:35:18.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7ff143fff640 1 --2- 172.21.15.161:0/1517147435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c006f20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff1380097a0 tx=0x7ff13802b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1913c190dbc3824a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f11dad9d640 1 --2- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc06d340 0x7f11dc0b3160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009d90 tx=0x7f11d402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a70fbbfbd7c0c1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f11d959a640 1 -- 172.21.15.161:0/2622242242 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d402cc50 con 0x7f11dc06d340 2024-03-23T14:35:18.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f11d959a640 1 -- 172.21.15.161:0/2622242242 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d402cdb0 con 0x7f11dc06d340 2024-03-23T14:35:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc726de9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7200d0850 0x7fc7200d0c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc726de9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc7200d1c20 0x7fc720166050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc726de9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc7200d49d0 0x7fc720166590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc726de9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc720162ae0 con 0x7fc7200d1c20 2024-03-23T14:35:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7f18c7c76640 1 -- start start 2024-03-23T14:35:18.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc726de9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc720163620 con 0x7fc7200d49d0 2024-03-23T14:35:18.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc726de9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc7200d1170 con 0x7fc7200d0850 2024-03-23T14:35:18.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7ff142ffd640 1 -- 172.21.15.161:0/1517147435 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138030710 con 0x7ff13c048100 2024-03-23T14:35:18.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7ff142ffd640 1 -- 172.21.15.161:0/1517147435 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff13802e050 con 0x7ff13c048100 2024-03-23T14:35:18.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7ff142ffd640 1 -- 172.21.15.161:0/1517147435 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff138030a90 con 0x7ff13c048100 2024-03-23T14:35:18.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f11d959a640 1 -- 172.21.15.161:0/2622242242 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4034660 con 0x7f11dc06d340 2024-03-23T14:35:18.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fa5fffc7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa5f80d0730 0x7fa5f80ceb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fa5fffc7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa5f815f0c0 0x7fa5f80d0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f18c7c76640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18c00d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f18c5a12640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f18c015e030 0x7f18c00ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33854/0 (socket says 172.21.15.161:33854) 2024-03-23T14:35:18.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.246+0000 7fb649009640 1 -- start start 2024-03-23T14:35:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f18c5a12640 1 -- 172.21.15.161:0/2501243397 learned_addr learned my addr 172.21.15.161:0/2501243397 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f18c7c76640 1 -- 172.21.15.161:0/2501243397 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00d0f90 con 0x7f18c00d0ae0 2024-03-23T14:35:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f18c7c76640 1 -- 172.21.15.161:0/2501243397 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18c00d2200 con 0x7f18c015e030 2024-03-23T14:35:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc725386640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc7200d49d0 0x7fc720166590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc725386640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc7200d49d0 0x7fc720166590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:38782/0 (socket says 172.21.15.161:38782) 2024-03-23T14:35:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fc725386640 1 -- 172.21.15.161:0/3629454920 learned_addr learned my addr 172.21.15.161:0/3629454920 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fa5fffc7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5f80d4c80 con 0x7fa5f8161bb0 2024-03-23T14:35:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fa5fffc7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa5f80d1050 con 0x7fa5f80d0730 2024-03-23T14:35:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fa5fffc7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa5f80cf0e0 con 0x7fa5f815f0c0 2024-03-23T14:35:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7f18c5211640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18c00d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c7c76640 1 -- 172.21.15.161:0/2501243397 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f18c00cec90 con 0x7f18c0161c20 2024-03-23T14:35:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa5fdd63640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa5f80d0730 0x7fa5f80ceb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa5fe564640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fa5f8161bb0 0x7fa5f8162000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33856/0 (socket says 172.21.15.161:33856) 2024-03-23T14:35:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa5fe564640 1 -- 172.21.15.161:0/3957487575 learned_addr learned my addr 172.21.15.161:0/3957487575 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fc71ffff640 1 --2- 172.21.15.161:0/3629454920 >> v2:172.21.15.161:3300/0 conn(0x7fc7200d1c20 0x7fc720166050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fc724b85640 1 --2- 172.21.15.161:0/3629454920 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7200d0850 0x7fc7200d0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fc725386640 1 -- 172.21.15.161:0/3629454920 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7200d0850 msgr2=0x7fc7200d0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fc725386640 1 --2- 172.21.15.161:0/3629454920 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7200d0850 0x7fc7200d0c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fc725386640 1 -- 172.21.15.161:0/3629454920 >> v2:172.21.15.161:3300/0 conn(0x7fc7200d1c20 msgr2=0x7fc720166050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fe6de9e2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe6d80b1650 0x7fe6d80af560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fe6de9e2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d80d0130 con 0x7fe6d802bfb0 2024-03-23T14:35:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fe6de9e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d80734b0 con 0x7fe6d8072cd0 2024-03-23T14:35:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fe6de9e2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fe6d80b1a30 con 0x7fe6d80b1650 2024-03-23T14:35:18.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3547071149 0 0) 0x7fa6500cf860 con 0x7fa65006d340 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa638003520 con 0x7fa65006d340 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.247+0000 7fa64f7fe640 1 --2- 172.21.15.161:0/3207661963 >> v2:172.21.15.161:3300/0 conn(0x7fa650071360 0x7fa65006ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2785493702 0 0) 0x7fa638003520 con 0x7fa65006d340 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 >> v2:172.21.15.183:3300/0 conn(0x7fa6500cf480 msgr2=0x7fa650163590 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 --2- 172.21.15.161:0/3207661963 >> v2:172.21.15.183:3300/0 conn(0x7fa6500cf480 0x7fa650163590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 >> v2:172.21.15.161:3300/0 conn(0x7fa650071360 msgr2=0x7fa65006ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 --2- 172.21.15.161:0/3207661963 >> v2:172.21.15.161:3300/0 conn(0x7fa650071360 0x7fa65006ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65006d770 con 0x7fa65006d340 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fa6480019f0 con 0x7fa65006d340 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 >> v1:172.21.15.161:6789/0 conn(0x7fa65006d340 legacy=0x7fa650166100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa638003970 0x7fa638003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6380047c0 0x7fa638004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 0x7fa63800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa65006f520 con 0x7fa6380047c0 2024-03-23T14:35:18.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa650072470 con 0x7fa638003970 2024-03-23T14:35:18.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa63800c220 con 0x7fa6380052a0 2024-03-23T14:35:18.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa5fdd63640 1 -- 172.21.15.161:0/3957487575 >> v2:172.21.15.183:3300/0 conn(0x7fa5f815f0c0 msgr2=0x7fa5f80d0b10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T14:35:18.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa5fdd63640 1 --2- 172.21.15.161:0/3957487575 >> v2:172.21.15.183:3300/0 conn(0x7fa5f815f0c0 0x7fa5f80d0b10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa5fdd63640 1 -- 172.21.15.161:0/3957487575 >> v1:172.21.15.183:6789/0 conn(0x7fa5f8161bb0 legacy=0x7fa5f8162000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 868843301 0 0) 0x7f18c00d2200 con 0x7f18c015e030 2024-03-23T14:35:18.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f18ac003520 con 0x7f18c015e030 2024-03-23T14:35:18.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fb649009640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb6441623b0 0x7fb6440cefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa5fdd63640 1 -- 172.21.15.161:0/3957487575 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f80cf3c0 con 0x7fa5f80d0730 2024-03-23T14:35:18.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fa6549cb640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 0x7fa63800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fe6dcf7f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe6d802bfb0 0x7fe6d815e240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49880/0 (socket says 172.21.15.161:49880) 2024-03-23T14:35:18.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2184456297 0 0) 0x7f18ac003520 con 0x7f18c015e030 2024-03-23T14:35:18.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 msgr2=0x7f18c00d05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c4a10640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18c00d05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 >> v1:172.21.15.161:6789/0 conn(0x7f18c00d0ae0 legacy=0x7f18c00d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fe6dcf7f640 1 -- 172.21.15.161:0/3238994687 learned_addr learned my addr 172.21.15.161:0/3238994687 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fb6437fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb6440c30f0 0x7fb644165dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33864/0 (socket says 172.21.15.161:33864) 2024-03-23T14:35:18.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fb6437fe640 1 -- 172.21.15.161:0/3826989054 learned_addr learned my addr 172.21.15.161:0/3826989054 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f11e0c15640 1 -- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc06d340 msgr2=0x7f11dc0b3160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f11e0c15640 1 --2- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc06d340 0x7f11dc0b3160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009d90 tx=0x7f11d402bea0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f11e0c15640 1 -- 172.21.15.161:0/2622242242 shutdown_connections 2024-03-23T14:35:18.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f11e0c15640 1 --2- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc06d340 0x7f11dc0b3160 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f11e0c15640 1 --2- 172.21.15.161:0/2622242242 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc071360 0x7f11dc06ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f11e0c15640 1 -- 172.21.15.161:0/2622242242 >> 172.21.15.161:0/2622242242 conn(0x7f11dc0b17f0 msgr2=0x7f11dc0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fa6549cb640 1 -- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa638003970 msgr2=0x7fa638003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fa6549cb640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa638003970 0x7fa638003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fa6549cb640 1 -- 172.21.15.161:0/3207661963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6380047c0 msgr2=0x7fa638004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fa6549cb640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6380047c0 0x7fa638004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fa6549cb640 1 -- 172.21.15.161:0/3207661963 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa65006d770 con 0x7fa6380052a0 2024-03-23T14:35:18.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d7fff640 1 --2- 172.21.15.161:0/3238994687 >> v2:172.21.15.183:3300/0 conn(0x7fe6d80b1650 0x7fe6d80af560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fb649009640 1 -- 172.21.15.161:0/3826989054 --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6440ce400 con 0x7fb6440c3560 2024-03-23T14:35:18.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c00cee80 con 0x7f18c015e030 2024-03-23T14:35:18.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f18b80019f0 con 0x7f18c015e030 2024-03-23T14:35:18.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 >> v1:172.21.15.183:6789/0 conn(0x7f18c015e030 legacy=0x7f18c00ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c4a10640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18ac003970 0x7f18ac003d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c4a10640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c4a10640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18ac004d00 0x7f18ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac014240 con 0x7f18ac004d00 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f18ac014380 con 0x7f18ac003970 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f18ac0144c0 con 0x7f18c0161c20 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fb649009640 1 -- 172.21.15.161:0/3826989054 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb644166be0 con 0x7fb6440c30f0 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 170295133 0 0) 0x7fe6d0001490 con 0x7fe6d802bfb0 2024-03-23T14:35:18.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6c0003520 con 0x7fe6d802bfb0 2024-03-23T14:35:18.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7ff14a888640 1 -- 172.21.15.161:0/1517147435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 msgr2=0x7ff13c006f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7ff14a888640 1 --2- 172.21.15.161:0/1517147435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c006f20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff1380097a0 tx=0x7ff13802b7b0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c5211640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fb643fff640 1 --2- 172.21.15.161:0/3826989054 >> v2:172.21.15.161:3300/0 conn(0x7fb6441623b0 0x7fb6440cefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7ff14a888640 1 -- 172.21.15.161:0/1517147435 shutdown_connections 2024-03-23T14:35:18.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7ff14a888640 1 --2- 172.21.15.161:0/1517147435 >> v2:172.21.15.161:3300/0 conn(0x7ff13c007460 0x7ff13c053e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7ff14a888640 1 --2- 172.21.15.161:0/1517147435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c006f20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7ff14a888640 1 -- 172.21.15.161:0/1517147435 >> 172.21.15.161:0/1517147435 conn(0x7ff13c009fe0 msgr2=0x7ff13c00a3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fb642ffd640 1 -- 172.21.15.161:0/3826989054 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3091749315 0 0) 0x7fb6440ce400 con 0x7fb6440c3560 2024-03-23T14:35:18.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7ff14a888640 1 -- 172.21.15.161:0/1517147435 shutdown_connections 2024-03-23T14:35:18.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fc725386640 1 --2- 172.21.15.161:0/3629454920 >> v2:172.21.15.161:3300/0 conn(0x7fc7200d1c20 0x7fc720166050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.248+0000 7fc725386640 1 -- 172.21.15.161:0/3629454920 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc720162cf0 con 0x7fc7200d49d0 2024-03-23T14:35:18.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fc71ffff640 1 --2- 172.21.15.161:0/3629454920 >> v2:172.21.15.161:3300/0 conn(0x7fc7200d1c20 0x7fc720166050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:18.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2381740167 0 0) 0x7fe6c0003520 con 0x7fe6d802bfb0 2024-03-23T14:35:18.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 >> v2:172.21.15.183:3300/0 conn(0x7fe6d80b1650 msgr2=0x7fe6d80af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 --2- 172.21.15.161:0/3238994687 >> v2:172.21.15.183:3300/0 conn(0x7fe6d80b1650 0x7fe6d80af560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 >> v1:172.21.15.183:6789/0 conn(0x7fe6d8072cd0 legacy=0x7fe6d815da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d80b11c0 con 0x7fe6d802bfb0 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fe6d0001b80 con 0x7fe6d802bfb0 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 >> v1:172.21.15.161:6789/0 conn(0x7fe6d802bfb0 legacy=0x7fe6d815e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6c00039c0 0x7fe6c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6c00047c0 0x7fe6c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6c000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d80b1a30 con 0x7fe6c00051d0 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe6c000c100 con 0x7fe6c00047c0 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe6c000c2c0 con 0x7fe6c00039c0 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c6213640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18ac004d00 0x7f18ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fe6dcf7f640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6c000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7ff14a888640 1 -- 172.21.15.161:0/1517147435 wait complete. 2024-03-23T14:35:18.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fe6d77fe640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6c00047c0 0x7fe6c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fe6d7fff640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6c00039c0 0x7fe6c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fe6dcf7f640 1 -- 172.21.15.161:0/3238994687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6c00039c0 msgr2=0x7fe6c0003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fe6dcf7f640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6c00039c0 0x7fe6c0003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fa6549cb640 1 -- 172.21.15.161:0/3207661963 reap_dead start 2024-03-23T14:35:18.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fa6549cb640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 0x7fa63800b980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa64800a990 tx=0x7fa6380057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb38670a1b0b47d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fe6dcf7f640 1 -- 172.21.15.161:0/3238994687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6c00047c0 msgr2=0x7fe6c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fe6dcf7f640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6c00047c0 0x7fe6c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fe6dcf7f640 1 -- 172.21.15.161:0/3238994687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6c0003520 con 0x7fe6c00051d0 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7fb649009640 1 -- 172.21.15.161:0/3826989054 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb64406e270 con 0x7fb6441623b0 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fb643fff640 1 -- 172.21.15.161:0/3826989054 >> v1:172.21.15.183:6789/0 conn(0x7fb6440c30f0 legacy=0x7fb644165dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fb643fff640 1 -- 172.21.15.161:0/3826989054 >> v1:172.21.15.161:6789/0 conn(0x7fb6440c3560 legacy=0x7fb6441664d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fb643fff640 1 -- 172.21.15.161:0/3826989054 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64406e550 con 0x7fb6441623b0 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c5211640 1 -- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18ac003970 msgr2=0x7f18ac003d70 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c5211640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18ac003970 0x7f18ac003d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c5211640 1 -- 172.21.15.161:0/2501243397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18ac004d00 msgr2=0x7f18ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c5211640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18ac004d00 0x7f18ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.249+0000 7f18c5211640 1 -- 172.21.15.161:0/2501243397 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18ac003520 con 0x7f18c0161c20 2024-03-23T14:35:18.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7f11e0c15640 1 -- 172.21.15.161:0/2622242242 shutdown_connections 2024-03-23T14:35:18.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fc725386640 1 --2- 172.21.15.161:0/3629454920 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7200d49d0 0x7fc720166590 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc720163590 tx=0x7fc71802bf00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b39cc59fa4748408 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fa5fdd63640 1 --2- 172.21.15.161:0/3957487575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f80d0730 0x7fa5f80ceb00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80d04f0 tx=0x7fa5f402b7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff695681bbbb5fb4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fc71f7fe640 1 -- 172.21.15.161:0/3629454920 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc71802caa0 con 0x7fc7200d49d0 2024-03-23T14:35:18.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fc71f7fe640 1 -- 172.21.15.161:0/3629454920 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc71802cc00 con 0x7fc7200d49d0 2024-03-23T14:35:18.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fc71f7fe640 1 -- 172.21.15.161:0/3629454920 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7180343f0 con 0x7fc7200d49d0 2024-03-23T14:35:18.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fa5fcd61640 1 -- 172.21.15.161:0/3957487575 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4031740 con 0x7fa5f80d0730 2024-03-23T14:35:18.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fa5fcd61640 1 -- 172.21.15.161:0/3957487575 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f40318a0 con 0x7fa5f80d0730 2024-03-23T14:35:18.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7f11e0c15640 1 -- 172.21.15.161:0/2622242242 wait complete. 2024-03-23T14:35:18.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7fa5fcd61640 1 -- 172.21.15.161:0/3957487575 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f4031740 con 0x7fa5f80d0730 2024-03-23T14:35:18.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648034630 con 0x7fa6380052a0 2024-03-23T14:35:18.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa64802f070 con 0x7fa6380052a0 2024-03-23T14:35:18.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.250+0000 7fa64effd640 1 -- 172.21.15.161:0/3207661963 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa648034a10 con 0x7fa6380052a0 2024-03-23T14:35:18.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7f18c5211640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18ac0047c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009ac0 tx=0x7f18ac0057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20a575ed1b2d024b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7fe6dcf7f640 1 -- 172.21.15.161:0/3238994687 reap_dead start 2024-03-23T14:35:18.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7fe6dcf7f640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6c000b980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000a730 tx=0x7fe6c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5d6d72ae043b69 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7ff14a888640 1 Processor -- start 2024-03-23T14:35:18.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d00072d0 con 0x7fe6c00051d0 2024-03-23T14:35:18.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6d002f070 con 0x7fe6c00051d0 2024-03-23T14:35:18.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7fe6d6ffd640 1 -- 172.21.15.161:0/3238994687 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6d0007670 con 0x7fe6c00051d0 2024-03-23T14:35:18.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7fb643fff640 1 --2- 172.21.15.161:0/3826989054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6441623b0 0x7fb6440cefc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009d90 tx=0x7fb63c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=993c9bf2cf54d5f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa5fffc7640 1 -- 172.21.15.161:0/3957487575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f80d0730 msgr2=0x7fa5f80ceb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa5fffc7640 1 --2- 172.21.15.161:0/3957487575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f80d0730 0x7fa5f80ceb00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80d04f0 tx=0x7fa5f402b7e0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.251+0000 7ff14a888640 1 -- start start 2024-03-23T14:35:18.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fb642ffd640 1 -- 172.21.15.161:0/3826989054 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c02cd10 con 0x7fb6441623b0 2024-03-23T14:35:18.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fb642ffd640 1 -- 172.21.15.161:0/3826989054 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb63c02ce70 con 0x7fb6441623b0 2024-03-23T14:35:18.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fb642ffd640 1 -- 172.21.15.161:0/3826989054 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c034640 con 0x7fb6441623b0 2024-03-23T14:35:18.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 -- 172.21.15.161:0/3629454920 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7200d49d0 msgr2=0x7fc720166590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 --2- 172.21.15.161:0/3629454920 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7200d49d0 0x7fc720166590 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc720163590 tx=0x7fc71802bf00 comp rx=0 tx=0).stop 2024-03-23T14:35:18.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa5fffc7640 1 -- 172.21.15.161:0/3957487575 shutdown_connections 2024-03-23T14:35:18.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa5fffc7640 1 --2- 172.21.15.161:0/3957487575 >> v2:172.21.15.183:3300/0 conn(0x7fa5f815f0c0 0x7fa5f80d0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa5fffc7640 1 --2- 172.21.15.161:0/3957487575 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f80d0730 0x7fa5f80ceb00 secure :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa5f80d04f0 tx=0x7fa5f402b7e0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa5fffc7640 1 -- 172.21.15.161:0/3957487575 >> 172.21.15.161:0/3957487575 conn(0x7fa5f80d3e40 msgr2=0x7fa5f80d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa65642e640 1 -- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 msgr2=0x7fa63800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa65642e640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 0x7fa63800b980 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fa64800a990 tx=0x7fa6380057f0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b002c710 con 0x7f18c0161c20 2024-03-23T14:35:18.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 -- 172.21.15.161:0/3629454920 shutdown_connections 2024-03-23T14:35:18.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 --2- 172.21.15.161:0/3629454920 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7200d49d0 0x7fc720166590 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 --2- 172.21.15.161:0/3629454920 >> v2:172.21.15.161:3300/0 conn(0x7fc7200d1c20 0x7fc720166050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 --2- 172.21.15.161:0/3629454920 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7200d0850 0x7fc7200d0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 -- 172.21.15.161:0/3629454920 >> 172.21.15.161:0/3629454920 conn(0x7fc7200d3e40 msgr2=0x7fc7200d4250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7f11e0c15640 1 Processor -- start 2024-03-23T14:35:18.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7f11e0c15640 1 -- start start 2024-03-23T14:35:18.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 -- 172.21.15.161:0/3629454920 shutdown_connections 2024-03-23T14:35:18.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa5fffc7640 1 -- 172.21.15.161:0/3957487575 shutdown_connections 2024-03-23T14:35:18.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fc726de9640 1 -- 172.21.15.161:0/3629454920 wait complete. 2024-03-23T14:35:18.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b002c870 con 0x7f18c0161c20 2024-03-23T14:35:18.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7f18c4a10640 1 -- 172.21.15.161:0/2501243397 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b002c710 con 0x7f18c0161c20 2024-03-23T14:35:18.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fe6de9e2640 1 -- 172.21.15.161:0/3238994687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 msgr2=0x7fe6c000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fe6de9e2640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6c000b980 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fe6d000a730 tx=0x7fe6c00057f0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fe6de9e2640 1 -- 172.21.15.161:0/3238994687 shutdown_connections 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fe6de9e2640 1 --2- 172.21.15.161:0/3238994687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6c000b980 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fe6de9e2640 1 -- 172.21.15.161:0/3238994687 >> 172.21.15.161:0/3238994687 conn(0x7fe6d80b5800 msgr2=0x7fe6d80725c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fe6de9e2640 1 -- 172.21.15.161:0/3238994687 shutdown_connections 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fe6de9e2640 1 -- 172.21.15.161:0/3238994687 wait complete. 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7ff14a888640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff13c007460 0x7ff13c0d8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7ff14a888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c049670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7ff14a888640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff13c054390 0x7ff13c035b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7ff14a888640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c034760 con 0x7ff13c007460 2024-03-23T14:35:18.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7ff14a888640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff13c037110 con 0x7ff13c054390 2024-03-23T14:35:18.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7ff14a888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff13c0349b0 con 0x7ff13c048100 2024-03-23T14:35:18.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c049670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7fb649009640 1 -- 172.21.15.161:0/3826989054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6441623b0 msgr2=0x7fb6440cefc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7fb649009640 1 --2- 172.21.15.161:0/3826989054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6441623b0 0x7fb6440cefc0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009d90 tx=0x7fb63c02bf60 comp rx=0 tx=0).stop 2024-03-23T14:35:18.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7ff1437fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c049670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42252/0 (socket says 172.21.15.161:42252) 2024-03-23T14:35:18.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7ff1437fe640 1 -- 172.21.15.161:0/1422743620 learned_addr learned my addr 172.21.15.161:0/1422743620 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7fb649009640 1 -- 172.21.15.161:0/3826989054 shutdown_connections 2024-03-23T14:35:18.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7fb649009640 1 --2- 172.21.15.161:0/3826989054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6441623b0 0x7fb6440cefc0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f18c7c76640 1 -- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 msgr2=0x7f18ac0047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f18c7c76640 1 --2- 172.21.15.161:0/2501243397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18ac0047c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f18b0009ac0 tx=0x7f18ac0057f0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f18c6213640 1 -- 172.21.15.161:0/2501243397 reap_dead start 2024-03-23T14:35:18.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7fb649009640 1 -- 172.21.15.161:0/3826989054 >> 172.21.15.161:0/3826989054 conn(0x7fb6440b1c50 msgr2=0x7fb6440b2030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7fb649009640 1 -- 172.21.15.161:0/3826989054 shutdown_connections 2024-03-23T14:35:18.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.252+0000 7fa5fffc7640 1 -- 172.21.15.161:0/3957487575 wait complete. 2024-03-23T14:35:18.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7fb649009640 1 -- 172.21.15.161:0/3826989054 wait complete. 2024-03-23T14:35:18.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f11e0c15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc06d340 0x7f11dc1fb0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f11e0c15640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc071360 0x7f11dc153510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f11e0c15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11dc0cf480 0x7f11dc1562c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f11e0c15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc145640 con 0x7f11dc0cf480 2024-03-23T14:35:18.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f11e0c15640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f11dc153ef0 con 0x7f11dc071360 2024-03-23T14:35:18.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7f11e0c15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f11dc1541d0 con 0x7f11dc06d340 2024-03-23T14:35:18.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.253+0000 7ff143fff640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff13c007460 0x7ff13c0d8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11dad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11dc0cf480 0x7f11dc1562c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f18c7c76640 1 -- 172.21.15.161:0/2501243397 shutdown_connections 2024-03-23T14:35:18.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f18c7c76640 1 -- 172.21.15.161:0/2501243397 >> 172.21.15.161:0/2501243397 conn(0x7f18c00b5a10 msgr2=0x7f18c00b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11da59c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc06d340 0x7f11dc1fb0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7fc726de9640 1 Processor -- start 2024-03-23T14:35:18.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11dad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11dc0cf480 0x7f11dc1562c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41134/0 (socket says 172.21.15.161:41134) 2024-03-23T14:35:18.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11dad9d640 1 -- 172.21.15.161:0/1778498170 learned_addr learned my addr 172.21.15.161:0/1778498170 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11d9d9b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc071360 0x7f11dc153510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7fa65642e640 1 -- 172.21.15.161:0/3207661963 shutdown_connections 2024-03-23T14:35:18.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7fa65642e640 1 --2- 172.21.15.161:0/3207661963 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 0x7fa63800b980 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7fa65642e640 1 -- 172.21.15.161:0/3207661963 >> 172.21.15.161:0/3207661963 conn(0x7fa6500b17f0 msgr2=0x7fa6500b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff1437fe640 1 -- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff13c054390 msgr2=0x7ff13c035b70 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-23T14:35:18.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff1437fe640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff13c054390 0x7ff13c035b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff1437fe640 1 -- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff13c007460 msgr2=0x7ff13c0d8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff1437fe640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff13c007460 0x7ff13c0d8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff1437fe640 1 -- 172.21.15.161:0/1422743620 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff138009400 con 0x7ff13c048100 2024-03-23T14:35:18.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11dad9d640 1 -- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc06d340 msgr2=0x7f11dc1fb0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11dad9d640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc06d340 0x7f11dc1fb0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7fa5fffc7640 1 Processor -- start 2024-03-23T14:35:18.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff143fff640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff13c007460 0x7ff13c0d8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:18.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff1437fe640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c049670 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff1440509c0 tx=0x7ff144068c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11dad9d640 1 -- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc071360 msgr2=0x7f11dc153510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11dad9d640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc071360 0x7f11dc153510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7fc726de9640 1 -- start start 2024-03-23T14:35:18.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11dad9d640 1 -- 172.21.15.161:0/1778498170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11cc009400 con 0x7f11dc0cf480 2024-03-23T14:35:18.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7fa5fffc7640 1 -- start start 2024-03-23T14:35:18.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff140ff9640 1 -- 172.21.15.161:0/1422743620 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144073280 con 0x7ff13c048100 2024-03-23T14:35:18.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff140ff9640 1 -- 172.21.15.161:0/1422743620 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1440733e0 con 0x7ff13c048100 2024-03-23T14:35:18.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7ff140ff9640 1 -- 172.21.15.161:0/1422743620 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff144071750 con 0x7ff13c048100 2024-03-23T14:35:18.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.254+0000 7f11da59c640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11dc06d340 0x7f11dc1fb0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:18.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7f11d9d9b640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11dc071360 0x7f11dc153510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:18.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff13c034c90 con 0x7ff13c048100 2024-03-23T14:35:18.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fe6de9e2640 1 Processor -- start 2024-03-23T14:35:18.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff13c037640 con 0x7ff13c048100 2024-03-23T14:35:18.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7f11dad9d640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11dc0cf480 0x7f11dc1562c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009eb0 tx=0x7f11d4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fb649009640 1 Processor -- start 2024-03-23T14:35:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fe6de9e2640 1 -- start start 2024-03-23T14:35:18.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc726de9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7200d0850 0x7fc72015c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc726de9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7200d49d0 0x7fc7201606b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc726de9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7201f8a50 0x7fc720160bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc726de9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc72015c7f0 con 0x7fc7200d0850 2024-03-23T14:35:18.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc726de9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc72015d9e0 con 0x7fc7201f8a50 2024-03-23T14:35:18.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc726de9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc72015dcc0 con 0x7fc7200d49d0 2024-03-23T14:35:18.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4034e40 con 0x7f11dc0cf480 2024-03-23T14:35:18.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11d4009a70 con 0x7f11dc0cf480 2024-03-23T14:35:18.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11dc154790 con 0x7f11dc0cf480 2024-03-23T14:35:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc724b85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7200d0850 0x7fc72015c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc724b85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7200d0850 0x7fc72015c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41140/0 (socket says 172.21.15.161:41140) 2024-03-23T14:35:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc724b85640 1 -- 172.21.15.161:0/325740384 learned_addr learned my addr 172.21.15.161:0/325740384 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fc725386640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7201f8a50 0x7fc720160bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fc724b85640 1 -- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7200d49d0 msgr2=0x7fc7201606b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-23T14:35:18.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fc724b85640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7200d49d0 0x7fc7201606b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fc724b85640 1 -- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7201f8a50 msgr2=0x7fc720160bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fc724b85640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7201f8a50 0x7fc720160bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fa5fffc7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f815f0c0 0x7fa5f80c9d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fc724b85640 1 -- 172.21.15.161:0/325740384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc718009b30 con 0x7fc7200d0850 2024-03-23T14:35:18.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11d403f420 con 0x7f11dc0cf480 2024-03-23T14:35:18.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fc724b85640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7200d0850 0x7fc72015c2b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc71400c7f0 tx=0x7fc71400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.255+0000 7fb649009640 1 -- start start 2024-03-23T14:35:18.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fe6de9e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6d81f6c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fb649009640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6440c30f0 0x7fb644159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7f18c7c76640 1 -- 172.21.15.161:0/2501243397 shutdown_connections 2024-03-23T14:35:18.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fe6de9e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6d814fdf0 0x7fe6d814c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fe6d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6d81f6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f11c0001030 con 0x7f11dc0cf480 2024-03-23T14:35:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb649009640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6440c3560 0x7fb6440d00f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714004220 con 0x7fc7200d0850 2024-03-23T14:35:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fe6d7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6d81f6c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41152/0 (socket says 172.21.15.161:41152) 2024-03-23T14:35:18.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb649009640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6441623b0 0x7fb6440d0630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fe6d7fff640 1 -- 172.21.15.161:0/2993968392 learned_addr learned my addr 172.21.15.161:0/2993968392 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fe6d77fe640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6d814fdf0 0x7fe6d814c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6440c3560 0x7fb6440d00f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb6437fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6440c3560 0x7fb6440d00f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42286/0 (socket says 172.21.15.161:42286) 2024-03-23T14:35:18.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fe6de9e2640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6d81f5210 0x7fe6d80b1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc714004380 con 0x7fc7200d0850 2024-03-23T14:35:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7200c7a10 con 0x7fc7200d0850 2024-03-23T14:35:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7200c7fd0 con 0x7fc7200d0850 2024-03-23T14:35:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb6437fe640 1 -- 172.21.15.161:0/348137443 learned_addr learned my addr 172.21.15.161:0/348137443 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb648808640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6441623b0 0x7fb6440d0630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7ff140ff9640 1 -- 172.21.15.161:0/1422743620 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff1440718b0 con 0x7ff13c048100 2024-03-23T14:35:18.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb6440c5b00 con 0x7fb6440c30f0 2024-03-23T14:35:18.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb6440c4ca0 con 0x7fb6441623b0 2024-03-23T14:35:18.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb6440c4ef0 con 0x7fb6440c3560 2024-03-23T14:35:18.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d81f7160 con 0x7fe6c00051d0 2024-03-23T14:35:18.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d814caf0 con 0x7fe6d81f5210 2024-03-23T14:35:18.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe6d81f5b80 con 0x7fe6d814fdf0 2024-03-23T14:35:18.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7ff140ff9640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff12806b810 0x7ff12806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7ff140ff9640 1 -- 172.21.15.161:0/1422743620 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff144104e90 con 0x7ff13c048100 2024-03-23T14:35:18.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7ff143fff640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff12806b810 0x7ff12806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc714007840 con 0x7fc7200d0850 2024-03-23T14:35:18.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7f18c7c76640 1 -- 172.21.15.161:0/2501243397 wait complete. 2024-03-23T14:35:18.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fe6dcf7f640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6d81f5210 0x7fe6d80b1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fe6d77fe640 1 -- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6d81f5210 msgr2=0x7fe6d80b1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fe6d77fe640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe6d81f5210 0x7fe6d80b1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fa5fffc7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5f8161bb0 0x7fa5f815ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fa5fffc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f815cfd0 0x7fa5f81f7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fa5fffc7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f80ca970 con 0x7fa5f815f0c0 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fa5fffc7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f815d460 con 0x7fa5f8161bb0 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fa5fffc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f80b9360 con 0x7fa5f815cfd0 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fe564640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f815cfd0 0x7fa5f81f7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fd562640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5f8161bb0 0x7fa5f815ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fe564640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f815cfd0 0x7fa5f81f7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42294/0 (socket says 172.21.15.161:42294) 2024-03-23T14:35:18.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fe564640 1 -- 172.21.15.161:0/265110857 learned_addr learned my addr 172.21.15.161:0/265110857 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fa65642e640 1 -- 172.21.15.161:0/3207661963 shutdown_connections 2024-03-23T14:35:18.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.256+0000 7fa65642e640 1 -- 172.21.15.161:0/3207661963 wait complete. 2024-03-23T14:35:18.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa65642e640 1 Processor -- start 2024-03-23T14:35:18.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc700001030 con 0x7fc7200d0850 2024-03-23T14:35:18.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fd562640 1 -- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f815cfd0 msgr2=0x7fa5f81f7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fd562640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f815cfd0 0x7fa5f81f7aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fe6d77fe640 1 -- 172.21.15.161:0/2993968392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 msgr2=0x7fe6d81f6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7f18c7c76640 1 Processor -- start 2024-03-23T14:35:18.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fd562640 1 -- 172.21.15.161:0/265110857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f815f0c0 msgr2=0x7fa5f80c9d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fd562640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f815f0c0 0x7fa5f80c9d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fe6d77fe640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe6c00051d0 0x7fe6d81f6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fd562640 1 -- 172.21.15.161:0/265110857 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f4009400 con 0x7fa5f8161bb0 2024-03-23T14:35:18.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7f18c7c76640 1 -- start start 2024-03-23T14:35:18.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fe6d77fe640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d81f5e60 con 0x7fe6d814fdf0 2024-03-23T14:35:18.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fb6437fe640 1 -- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6441623b0 msgr2=0x7fb6440d0630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fb6437fe640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb6441623b0 0x7fb6440d0630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fb6437fe640 1 -- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6440c30f0 msgr2=0x7fb644159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa5fe564640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f815cfd0 0x7fa5f81f7aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:18.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fe6d77fe640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6d814fdf0 0x7fe6d814c5b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00f570 tx=0x7fe6cc00fa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc007260 con 0x7fe6d814fdf0 2024-03-23T14:35:18.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe6cc0073c0 con 0x7fe6d814fdf0 2024-03-23T14:35:18.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.258+0000 7fa65642e640 1 -- start start 2024-03-23T14:35:18.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d81f6140 con 0x7fe6d814fdf0 2024-03-23T14:35:18.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6cc0067d0 con 0x7fe6d814fdf0 2024-03-23T14:35:18.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa5fd562640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5f8161bb0 0x7fa5f815ca90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800b410 tx=0x7fa5e800b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d81f6700 con 0x7fe6d814fdf0 2024-03-23T14:35:18.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7ff143fff640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff12806b810 0x7ff12806dcd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff138034000 tx=0x7ff138008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7ff14a888640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7ff124001650 0x7ff124003b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 --> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff124006bb0 con 0x7ff124001650 2024-03-23T14:35:18.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7ff148e25640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7ff124001650 0x7ff124003b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.257+0000 7fb643fff640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6440c30f0 0x7fb644159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fb6437fe640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6440c30f0 0x7fb644159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fb6437fe640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb63c009a70 con 0x7fb6440c3560 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fb643fff640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb6440c30f0 0x7fb644159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa65642e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 0x7fa650156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa65642e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6501587e0 0x7fa6500af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa65642e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6501639f0 0x7fa650163e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa65642e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa650157690 con 0x7fa6501587e0 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa65642e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa650156b10 con 0x7fa6501639f0 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa65642e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa650156df0 con 0x7fa6380052a0 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6501587e0 0x7fa6500af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fa64f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6501587e0 0x7fa6500af9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41182/0 (socket says 172.21.15.161:41182) 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fa64f7fe640 1 -- 172.21.15.161:0/1575628524 learned_addr learned my addr 172.21.15.161:0/1575628524 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fa6549cb640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6501639f0 0x7fa650163e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fa64ffff640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 0x7fa650156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8011830 con 0x7fa5f8161bb0 2024-03-23T14:35:18.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.259+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f80b8470 con 0x7fa5f8161bb0 2024-03-23T14:35:18.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f80b8a30 con 0x7fa5f8161bb0 2024-03-23T14:35:18.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c7c76640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18c00c6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c7c76640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18c00b53b0 0x7f18c00c5b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c5a12640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18c00c6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11d4034e40 con 0x7f11dc0cf480 2024-03-23T14:35:18.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f11d403f590 con 0x7f11dc0cf480 2024-03-23T14:35:18.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fb6437fe640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6440c3560 0x7fb6440d00f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb63800c480 tx=0x7fb63800c940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c5a12640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18c00c6760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42312/0 (socket says 172.21.15.161:42312) 2024-03-23T14:35:18.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c5a12640 1 -- 172.21.15.161:0/1217034525 learned_addr learned my addr 172.21.15.161:0/1217034525 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c7c76640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18c014a960 0x7f18c01558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c5211640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18c00b53b0 0x7f18c00c5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7ff148e25640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7ff124001650 0x7ff124003b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe6d80b0e50 con 0x7fe6d814fdf0 2024-03-23T14:35:18.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c7c76640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f18c0157050 con 0x7f18c00b53b0 2024-03-23T14:35:18.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f18c6213640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18c014a960 0x7f18c01558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7f18c7c76640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f18c0156a90 con 0x7f18c014a960 2024-03-23T14:35:18.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7f18c7c76640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f18c0156d70 con 0x7f18c0161c20 2024-03-23T14:35:18.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5e8011990 con 0x7fa5f8161bb0 2024-03-23T14:35:18.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7f11caffd640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f11bc06b820 0x7f11bc06dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f11d40ace40 con 0x7f11dc0cf480 2024-03-23T14:35:18.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7f11caffd640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f11bc073aa0 0x7f11bc075ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 --> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11bc076570 con 0x7f11bc073aa0 2024-03-23T14:35:18.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f11d40ad1a0 con 0x7f11dc0cf480 2024-03-23T14:35:18.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7f11d9d9b640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f11bc073aa0 0x7f11bc075ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8011d40 con 0x7fa5f8161bb0 2024-03-23T14:35:18.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa5f80cfc60 con 0x7fa5f8161bb0 2024-03-23T14:35:18.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe6cc006930 con 0x7fe6d814fdf0 2024-03-23T14:35:18.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb638005ce0 con 0x7fb6440c3560 2024-03-23T14:35:18.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb638005e40 con 0x7fb6440c3560 2024-03-23T14:35:18.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.260+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6380072f0 con 0x7fb6440c3560 2024-03-23T14:35:18.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7ff140ff9640 1 -- 172.21.15.161:0/1422743620 <== osd.0 v2:172.21.15.161:6820/3143302717 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff124006bb0 con 0x7ff124001650 2024-03-23T14:35:18.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6440c51d0 con 0x7fb6440c3560 2024-03-23T14:35:18.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6440c5790 con 0x7fb6440c3560 2024-03-23T14:35:18.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fa6549cb640 1 -- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 msgr2=0x7fa650156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fa6549cb640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa6380052a0 0x7fa650156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fa6549cb640 1 -- 172.21.15.161:0/1575628524 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6501587e0 msgr2=0x7fa6500af9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fa6549cb640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa6501587e0 0x7fa6500af9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.261+0000 7fa6549cb640 1 -- 172.21.15.161:0/1575628524 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6501570d0 con 0x7fa6501639f0 2024-03-23T14:35:18.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fe6d4ff9640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe6c406b890 0x7fe6c406dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe6cc089e80 con 0x7fe6d814fdf0 2024-03-23T14:35:18.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fe6d4ff9640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7fe6c4073a30 0x7fe6c4075e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6c4076500 con 0x7fe6c4073a30 2024-03-23T14:35:18.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f11da59c640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f11bc06b820 0x7f11bc06dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fe6cc054890 con 0x7fe6d814fdf0 2024-03-23T14:35:18.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fe6dcf7f640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7fe6c4073a30 0x7fe6c4075e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f11d9d9b640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f11bc073aa0 0x7f11bc075ec0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f11da59c640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f11bc06b820 0x7f11bc06dce0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11d0000fb0 tx=0x7f11d0007d70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc714007bb0 con 0x7fc7200d0850 2024-03-23T14:35:18.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fc71d7fa640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc70c06b8e0 0x7fc70c06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb64406edf0 con 0x7fb6440c3560 2024-03-23T14:35:18.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc71408a620 con 0x7fc7200d0850 2024-03-23T14:35:18.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fc71d7fa640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fc70c073b00 0x7fc70c075f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fc71ffff640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc70c06b8e0 0x7fc70c06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 --> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc70c0765d0 con 0x7fc70c073b00 2024-03-23T14:35:18.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fc71408a9d0 con 0x7fc7200d0850 2024-03-23T14:35:18.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fa6549cb640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6501639f0 0x7fa650163e50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6500d0b40 tx=0x7fa648001cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fe6d7fff640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe6c406b890 0x7fe6c406dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.324 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fc725386640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fc70c073b00 0x7fc70c075f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480345f0 con 0x7fa6501639f0 2024-03-23T14:35:18.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6480019c0 con 0x7fa6501639f0 2024-03-23T14:35:18.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480349f0 con 0x7fa6501639f0 2024-03-23T14:35:18.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fc71ffff640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc70c06b8e0 0x7fc70c06dda0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc7200c9d20 tx=0x7fc710008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb638007510 con 0x7fb6440c3560 2024-03-23T14:35:18.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7fb640ff9640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb63006b8e0 0x7fb63006dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 <== osd.2 v2:172.21.15.161:6809/4063400408 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f11bc076570 con 0x7f11bc073aa0 2024-03-23T14:35:18.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fc725386640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fc70c073b00 0x7fc70c075f20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fa65642e640 1 -- 172.21.15.161:0/1575628524 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6501573b0 con 0x7fa6501639f0 2024-03-23T14:35:18.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7fa65642e640 1 -- 172.21.15.161:0/1575628524 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6500aff10 con 0x7fa6501639f0 2024-03-23T14:35:18.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa5e801a400 con 0x7fa5f8161bb0 2024-03-23T14:35:18.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7fe6dcf7f640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7fe6c4073a30 0x7fe6c4075e50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f18c5211640 1 -- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 msgr2=0x7f18c00c6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f18c5211640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18c0161c20 0x7f18c00c6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f18c5211640 1 -- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18c014a960 msgr2=0x7f18c01558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f18c5211640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18c014a960 0x7f18c01558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f18c5211640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c01553b0 con 0x7f18c00b53b0 2024-03-23T14:35:18.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.262+0000 7f18c5211640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18c00b53b0 0x7f18c00c5b60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18b0033000 tx=0x7f18b0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b002c710 con 0x7f18c00b53b0 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18b002cbf0 con 0x7f18c00b53b0 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18b0003610 con 0x7f18c00b53b0 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7f18c7c76640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18c00c60a0 con 0x7f18c00b53b0 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.263+0000 7f18c7c76640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18c014ae10 con 0x7f18c00b53b0 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7f18c6213640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18c014a960 0x7f18c01558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb6380896e0 con 0x7fb6440c3560 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7fb640ff9640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7fb630073a80 0x7fb630075ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb630076550 con 0x7fb630073a80 2024-03-23T14:35:18.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fb638001160 con 0x7fb6440c3560 2024-03-23T14:35:18.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.264+0000 7fb643fff640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb63006b8e0 0x7fb63006dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fa5ee7fc640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5dc06b8e0 0x7fa5dc06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 <== osd.3 v2:172.21.15.161:6800/162554479 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc70c0765d0 con 0x7fc70c073b00 2024-03-23T14:35:18.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fb648808640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7fb630073a80 0x7fb630075ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa650071e20 con 0x7fa6501639f0 2024-03-23T14:35:18.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa5e8089f00 con 0x7fa5f8161bb0 2024-03-23T14:35:18.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fa5fdd63640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5dc06b8e0 0x7fa5dc06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fb643fff640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb63006b8e0 0x7fb63006dda0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb63c036000 tx=0x7fb63c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fb648808640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7fb630073a80 0x7fb630075ea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fa5ee7fc640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa5dc073990 0x7fa5dc075db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 --> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5dc076460 con 0x7fa5dc073990 2024-03-23T14:35:18.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa5e808c520 con 0x7fa5f8161bb0 2024-03-23T14:35:18.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 <== osd.6 v2:172.21.15.183:6800/963839040 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe6c4076500 con 0x7fe6c4073a30 2024-03-23T14:35:18.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7fa5fe564640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa5dc073990 0x7fa5dc075db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.265+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f18b0003770 con 0x7f18c00b53b0 2024-03-23T14:35:18.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7fe6d7fff640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe6c406b890 0x7fe6c406dd50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8006d40 tx=0x7fe6c8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7fa5fdd63640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5dc06b8e0 0x7fa5dc06dda0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4035000 tx=0x7fa5f4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f18c00d24c0 con 0x7f18c00b53b0 2024-03-23T14:35:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7f18b67fc640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18a406b910 0x7f18a406ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f18b00ac140 con 0x7f18c00b53b0 2024-03-23T14:35:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7f18b67fc640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7f18a4073b10 0x7f18a4075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18a40765e0 con 0x7f18a4073b10 2024-03-23T14:35:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7f18c5a12640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18a406b910 0x7f18a406ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7f18b0076c80 con 0x7f18c00b53b0 2024-03-23T14:35:18.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.266+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa648031030 con 0x7fa6501639f0 2024-03-23T14:35:18.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7fa5fe564640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa5dc073990 0x7fa5dc075db0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7f18c6213640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7f18a4073b10 0x7f18a4075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7f18c5a12640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18a406b910 0x7f18a406ddd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18b8005fd0 tx=0x7f18b8001a90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7fa64cff9640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa63c06b810 0x7fa63c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7fa64ffff640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa63c06b810 0x7fa63c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa6480ab540 con 0x7fa6501639f0 2024-03-23T14:35:18.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7fa64cff9640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7fa63c073a10 0x7fa63c075e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 --> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa63c0764e0 con 0x7fa63c073a10 2024-03-23T14:35:18.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.267+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (secure 0 0 0) 0x7fa6480ab990 con 0x7fa6501639f0 2024-03-23T14:35:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.268+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 <== osd.5 v2:172.21.15.183:6816/2304513884 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa5dc076460 con 0x7fa5dc073990 2024-03-23T14:35:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.268+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 <== osd.1 v2:172.21.15.161:6808/2842040386 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb630076550 con 0x7fb630073a80 2024-03-23T14:35:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.268+0000 7fa64f7fe640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7fa63c073a10 0x7fa63c075e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.268+0000 7fa64ffff640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa63c06b810 0x7fa63c06dcd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa640009900 tx=0x7fa6400090e0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.268+0000 7f18c6213640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7f18a4073b10 0x7f18a4075f30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.268+0000 7fa64f7fe640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7fa63c073a10 0x7fa63c075e30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.269+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 <== osd.7 v2:172.21.15.183:6808/4120799670 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f18a40765e0 con 0x7f18a4073b10 2024-03-23T14:35:18.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.269+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 <== osd.4 v2:172.21.15.183:6820/3154566470 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa63c0764e0 con 0x7fa63c073a10 2024-03-23T14:35:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.293+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 --> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff124005c90 con 0x7ff124001650 2024-03-23T14:35:18.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.293+0000 7ff140ff9640 1 -- 172.21.15.161:0/1422743620 <== osd.0 v2:172.21.15.161:6820/3143302717 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7ff124005c90 con 0x7ff124001650 2024-03-23T14:35:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.294+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7ff124001650 msgr2=0x7ff124003b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.294+0000 7ff14a888640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7ff124001650 0x7ff124003b10 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.294+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff12806b810 msgr2=0x7ff12806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.294+0000 7ff14a888640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff12806b810 0x7ff12806dcd0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff138034000 tx=0x7ff138008030 comp rx=0 tx=0).stop 2024-03-23T14:35:18.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.294+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 msgr2=0x7ff13c049670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.295+0000 7ff14a888640 1 --2- 172.21.15.161:0/1422743620 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff13c048100 0x7ff13c049670 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ff1440509c0 tx=0x7ff144068c40 comp rx=0 tx=0).stop 2024-03-23T14:35:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.295+0000 7ff148e25640 1 -- 172.21.15.161:0/1422743620 reap_dead start 2024-03-23T14:35:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.295+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 shutdown_connections 2024-03-23T14:35:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.295+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 >> 172.21.15.161:0/1422743620 conn(0x7ff13c009fe0 msgr2=0x7ff13c0e6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.295+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 shutdown_connections 2024-03-23T14:35:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.295+0000 7ff14a888640 1 -- 172.21.15.161:0/1422743620 wait complete. 2024-03-23T14:35:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.296+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 --> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc700002d30 con 0x7fc70c073b00 2024-03-23T14:35:18.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.297+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 --> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f11c0002d30 con 0x7f11bc073aa0 2024-03-23T14:35:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.297+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 --> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe6d81f5870 con 0x7fe6c4073a30 2024-03-23T14:35:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.297+0000 7fc71d7fa640 1 -- 172.21.15.161:0/325740384 <== osd.3 v2:172.21.15.161:6800/162554479 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fc700002d30 con 0x7fc70c073b00 2024-03-23T14:35:18.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.297+0000 7fe6d4ff9640 1 -- 172.21.15.161:0/2993968392 <== osd.6 v2:172.21.15.183:6800/963839040 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe6d81f5870 con 0x7fe6c4073a30 2024-03-23T14:35:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.297+0000 7f11caffd640 1 -- 172.21.15.161:0/1778498170 <== osd.2 v2:172.21.15.161:6809/4063400408 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f11c0002d30 con 0x7f11bc073aa0 2024-03-23T14:35:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fc70c073b00 msgr2=0x7fc70c075f20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc726de9640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fc70c073b00 0x7fc70c075f20 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc70c06b8e0 msgr2=0x7fc70c06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc726de9640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc70c06b8e0 0x7fc70c06dda0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc7200c9d20 tx=0x7fc710008040 comp rx=0 tx=0).stop 2024-03-23T14:35:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f11bc073aa0 msgr2=0x7f11bc075ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7200d0850 msgr2=0x7fc72015c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7f11e0c15640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f11bc073aa0 0x7f11bc075ec0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc726de9640 1 --2- 172.21.15.161:0/325740384 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7200d0850 0x7fc72015c2b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc71400c7f0 tx=0x7fc71400ccb0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7fe6c4073a30 msgr2=0x7fe6c4075e50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6de9e2640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7fe6c4073a30 0x7fe6c4075e50 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe6c406b890 msgr2=0x7fe6c406dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6de9e2640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe6c406b890 0x7fe6c406dd50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8006d40 tx=0x7fe6c8008040 comp rx=0 tx=0).stop 2024-03-23T14:35:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6d814fdf0 msgr2=0x7fe6d814c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6de9e2640 1 --2- 172.21.15.161:0/2993968392 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe6d814fdf0 0x7fe6d814c5b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe6cc00f570 tx=0x7fe6cc00fa30 comp rx=0 tx=0).stop 2024-03-23T14:35:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6dcf7f640 1 -- 172.21.15.161:0/2993968392 reap_dead start 2024-03-23T14:35:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 shutdown_connections 2024-03-23T14:35:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 >> 172.21.15.161:0/2993968392 conn(0x7fe6d80b5800 msgr2=0x7fe6d80b3230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc725386640 1 -- 172.21.15.161:0/325740384 reap_dead start 2024-03-23T14:35:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 shutdown_connections 2024-03-23T14:35:18.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.298+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 >> 172.21.15.161:0/325740384 conn(0x7fc7200d3e40 msgr2=0x7fc7200b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f11bc06b820 msgr2=0x7f11bc06dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11e0c15640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f11bc06b820 0x7f11bc06dce0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f11d0000fb0 tx=0x7f11d0007d70 comp rx=0 tx=0).stop 2024-03-23T14:35:18.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11dc0cf480 msgr2=0x7f11dc1562c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11e0c15640 1 --2- 172.21.15.161:0/1778498170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f11dc0cf480 0x7f11dc1562c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f11d4009eb0 tx=0x7f11d4004a40 comp rx=0 tx=0).stop 2024-03-23T14:35:18.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11dad9d640 1 -- 172.21.15.161:0/1778498170 reap_dead start 2024-03-23T14:35:18.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 shutdown_connections 2024-03-23T14:35:18.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 >> 172.21.15.161:0/1778498170 conn(0x7f11dc0b17f0 msgr2=0x7f11dc164750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 shutdown_connections 2024-03-23T14:35:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 shutdown_connections 2024-03-23T14:35:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7fc726de9640 1 -- 172.21.15.161:0/325740384 wait complete. 2024-03-23T14:35:18.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7f11e0c15640 1 -- 172.21.15.161:0/1778498170 wait complete. 2024-03-23T14:35:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 shutdown_connections 2024-03-23T14:35:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 --> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb64406eb10 con 0x7fb630073a80 2024-03-23T14:35:18.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7fe6de9e2640 1 -- 172.21.15.161:0/2993968392 wait complete. 2024-03-23T14:35:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.299+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 --> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa5f80cf980 con 0x7fa5dc073990 2024-03-23T14:35:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb640ff9640 1 -- 172.21.15.161:0/348137443 <== osd.1 v2:172.21.15.161:6808/2842040386 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fb634001400 con 0x7fb630073a80 2024-03-23T14:35:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fa5ee7fc640 1 -- 172.21.15.161:0/265110857 <== osd.5 v2:172.21.15.183:6816/2304513884 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa5f000f040 con 0x7fa5dc073990 2024-03-23T14:35:18.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 --> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f18c00cf440 con 0x7f18a4073b10 2024-03-23T14:35:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7fb630073a80 msgr2=0x7fb630075ea0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7f18b67fc640 1 -- 172.21.15.161:0/1217034525 <== osd.7 v2:172.21.15.183:6808/4120799670 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f18a40765e0 con 0x7f18a4073b10 2024-03-23T14:35:18.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb649009640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7fb630073a80 0x7fb630075ea0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb63006b8e0 msgr2=0x7fb63006dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb649009640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb63006b8e0 0x7fb63006dda0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb63c036000 tx=0x7fb63c030000 comp rx=0 tx=0).stop 2024-03-23T14:35:18.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6440c3560 msgr2=0x7fb6440d00f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb649009640 1 --2- 172.21.15.161:0/348137443 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb6440c3560 0x7fb6440d00f0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fb63800c480 tx=0x7fb63800c940 comp rx=0 tx=0).stop 2024-03-23T14:35:18.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb648808640 1 -- 172.21.15.161:0/348137443 reap_dead start 2024-03-23T14:35:18.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 shutdown_connections 2024-03-23T14:35:18.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.300+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 >> 172.21.15.161:0/348137443 conn(0x7fb6440b1c50 msgr2=0x7fb64406d020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 shutdown_connections 2024-03-23T14:35:18.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fb649009640 1 -- 172.21.15.161:0/348137443 wait complete. 2024-03-23T14:35:18.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 --> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa6501645e0 con 0x7fa63c073a10 2024-03-23T14:35:18.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa5dc073990 msgr2=0x7fa5dc075db0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa5fffc7640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa5dc073990 0x7fa5dc075db0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7f18a4073b10 msgr2=0x7f18a4075f30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7f18a3fff640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7f18a4073b10 0x7f18a4075f30 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5dc06b8e0 msgr2=0x7fa5dc06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa5fffc7640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5dc06b8e0 0x7fa5dc06dda0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fa5f4035000 tx=0x7fa5f4030000 comp rx=0 tx=0).stop 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5f8161bb0 msgr2=0x7fa5f815ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa5fffc7640 1 --2- 172.21.15.161:0/265110857 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5f8161bb0 0x7fa5f815ca90 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fa5e800b410 tx=0x7fa5e800b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa5fe564640 1 -- 172.21.15.161:0/265110857 reap_dead start 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18a406b910 msgr2=0x7f18a406ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7f18a3fff640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18a406b910 0x7f18a406ddd0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18b8005fd0 tx=0x7f18b8001a90 comp rx=0 tx=0).stop 2024-03-23T14:35:18.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18c00b53b0 msgr2=0x7f18c00c5b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7f18a3fff640 1 --2- 172.21.15.161:0/1217034525 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18c00b53b0 0x7f18c00c5b60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f18b0033000 tx=0x7f18b0004730 comp rx=0 tx=0).stop 2024-03-23T14:35:18.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7f18c6213640 1 -- 172.21.15.161:0/1217034525 reap_dead start 2024-03-23T14:35:18.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.301+0000 7fa64cff9640 1 -- 172.21.15.161:0/1575628524 <== osd.4 v2:172.21.15.183:6820/3154566470 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa6501645e0 con 0x7fa63c073a10 2024-03-23T14:35:18.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.302+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 shutdown_connections 2024-03-23T14:35:18.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.302+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 >> 172.21.15.161:0/265110857 conn(0x7fa5f80d3e40 msgr2=0x7fa5f815e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.302+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 shutdown_connections 2024-03-23T14:35:18.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.302+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 >> 172.21.15.161:0/1217034525 conn(0x7f18c00b5a10 msgr2=0x7f18c0151460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.302+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 shutdown_connections 2024-03-23T14:35:18.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.302+0000 7fa5fffc7640 1 -- 172.21.15.161:0/265110857 wait complete. 2024-03-23T14:35:18.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 shutdown_connections 2024-03-23T14:35:18.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7f18a3fff640 1 -- 172.21.15.161:0/1217034525 wait complete. 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7fa63c073a10 msgr2=0x7fa63c075e30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7fa6367fc640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7fa63c073a10 0x7fa63c075e30 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa63c06b810 msgr2=0x7fa63c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7fa6367fc640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa63c06b810 0x7fa63c06dcd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa640009900 tx=0x7fa6400090e0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6501639f0 msgr2=0x7fa650163e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7fa6367fc640 1 --2- 172.21.15.161:0/1575628524 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa6501639f0 0x7fa650163e50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fa6500d0b40 tx=0x7fa648001cb0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.303+0000 7fa6549cb640 1 -- 172.21.15.161:0/1575628524 reap_dead start 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.304+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 shutdown_connections 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.304+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 >> 172.21.15.161:0/1575628524 conn(0x7fa6500b17f0 msgr2=0x7fa65015ffb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.304+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 shutdown_connections 2024-03-23T14:35:18.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.304+0000 7fa6367fc640 1 -- 172.21.15.161:0/1575628524 wait complete. 2024-03-23T14:35:18.358 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-23T14:35:18.358 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-03-23T14:35:18.359 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-23T14:35:18.359 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-03-23T14:35:18.360 INFO:teuthology.orchestra.run.smithi161.stdout:55834574854 2024-03-23T14:35:18.360 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-23T14:35:18.361 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-03-23T14:35:18.361 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-03-23T14:35:18.362 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-23T14:35:18.363 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-03-23T14:35:18.363 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-23T14:35:18.364 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-03-23T14:35:18.364 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-23T14:35:18.365 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-03-23T14:35:18.365 INFO:teuthology.orchestra.run.smithi161.stdout:55834574854 2024-03-23T14:35:18.366 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-03-23T14:35:18.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.573+0000 7efed8cae640 1 Processor -- start 2024-03-23T14:35:18.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.573+0000 7efed8cae640 1 -- start start 2024-03-23T14:35:18.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7fc280dc7640 1 Processor -- start 2024-03-23T14:35:18.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7fc280dc7640 1 -- start start 2024-03-23T14:35:18.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7fa972bc2640 1 Processor -- start 2024-03-23T14:35:18.570 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7fa972bc2640 1 -- start start 2024-03-23T14:35:18.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7efed8cae640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efed40cd890 0x7efed416cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7efed8cae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed40cf470 con 0x7efed40c4b40 2024-03-23T14:35:18.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7efed8cae640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efed40cc3f0 con 0x7efed40d2690 2024-03-23T14:35:18.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7efed8cae640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7efed40cebd0 con 0x7efed40cd890 2024-03-23T14:35:18.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.574+0000 7f265776c640 1 Processor -- start 2024-03-23T14:35:18.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7efed2d9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efed40cd890 0x7efed416cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7efed2d9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efed40cd890 0x7efed416cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:38898/0 (socket says 172.21.15.161:38898) 2024-03-23T14:35:18.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7efed2d9d640 1 -- 172.21.15.161:0/3272758786 learned_addr learned my addr 172.21.15.161:0/3272758786 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7efed259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efed40c4b40 0x7efed40ce3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49894/0 (socket says 172.21.15.161:49894) 2024-03-23T14:35:18.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7f265776c640 1 -- start start 2024-03-23T14:35:18.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7efed2d9d640 1 -- 172.21.15.161:0/3272758786 >> v1:172.21.15.183:6789/0 conn(0x7efed40d2690 legacy=0x7efed40c4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7efed2d9d640 1 -- 172.21.15.161:0/3272758786 >> v1:172.21.15.161:6789/0 conn(0x7efed40c4b40 legacy=0x7efed40ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7fc280dc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc274038070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7fa972bc2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa96c15da90 0x7fa96c0b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f145e9d7640 1 Processor -- start 2024-03-23T14:35:18.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f145e9d7640 1 -- start start 2024-03-23T14:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7fc280dc7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc274053c60 0x7fc2740df210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7fc280dc7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc2740df750 con 0x7fc274033ee0 2024-03-23T14:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7fc280dc7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc2740544b0 con 0x7fc274053c60 2024-03-23T14:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.575+0000 7fc280dc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc2740541d0 con 0x7fc27400c090 2024-03-23T14:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc27ad9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc274033ee0 0x7fc27404fa80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33892/0 (socket says 172.21.15.161:33892) 2024-03-23T14:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc27ad9d640 1 -- 172.21.15.161:0/4123644765 learned_addr learned my addr 172.21.15.161:0/4123644765 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc27a59c640 1 --2- 172.21.15.161:0/4123644765 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc274038070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc279d9b640 1 --2- 172.21.15.161:0/4123644765 >> v2:172.21.15.161:3300/0 conn(0x7fc274053c60 0x7fc2740df210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fe0ea90a640 1 Processor -- start 2024-03-23T14:35:18.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa972bc2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0cf5c0 0x7fa96c0b3560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f265776c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2650145080 0x7f2650072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa972bc2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa96c0b3f00 con 0x7fa96c0b3aa0 2024-03-23T14:35:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa972bc2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa96c06ed30 con 0x7fa96c15da90 2024-03-23T14:35:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa972bc2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa96c15ece0 con 0x7fa96c0cf5c0 2024-03-23T14:35:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa97095e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa96c15da90 0x7fa96c0b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa97095e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa96c15da90 0x7fa96c0b4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41204/0 (socket says 172.21.15.161:41204) 2024-03-23T14:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa97095e640 1 -- 172.21.15.161:0/302124871 learned_addr learned my addr 172.21.15.161:0/302124871 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f265776c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f26501607d0 0x7f265015fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc27a59c640 1 -- 172.21.15.161:0/4123644765 >> v2:172.21.15.161:3300/0 conn(0x7fc274053c60 msgr2=0x7fc2740df210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc27a59c640 1 --2- 172.21.15.161:0/4123644765 >> v2:172.21.15.161:3300/0 conn(0x7fc274053c60 0x7fc2740df210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc27a59c640 1 -- 172.21.15.161:0/4123644765 >> v1:172.21.15.183:6789/0 conn(0x7fc274033ee0 legacy=0x7fc27404fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc27a59c640 1 -- 172.21.15.161:0/4123644765 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc274005f20 con 0x7fc27400c090 2024-03-23T14:35:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fc279d9b640 1 --2- 172.21.15.161:0/4123644765 >> v2:172.21.15.161:3300/0 conn(0x7fc274053c60 0x7fc2740df210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:18.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa96bfff640 1 --2- 172.21.15.161:0/302124871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0cf5c0 0x7fa96c0b3560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f265776c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26500cf4f0 con 0x7f265006cbf0 2024-03-23T14:35:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f265776c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2650071010 con 0x7f26501607d0 2024-03-23T14:35:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f265776c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f26500cef30 con 0x7f2650145080 2024-03-23T14:35:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f2655508640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2650145080 0x7f2650072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f2655d09640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f265006cbf0 0x7f2650071430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49898/0 (socket says 172.21.15.161:49898) 2024-03-23T14:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7f2655d09640 1 -- 172.21.15.161:0/246799386 learned_addr learned my addr 172.21.15.161:0/246799386 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f145e9d7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1450035f70 0x7f1450057630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7efed2d9d640 1 -- 172.21.15.161:0/3272758786 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efed40ceeb0 con 0x7efed40cd890 2024-03-23T14:35:18.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7efed2d9d640 1 --2- 172.21.15.161:0/3272758786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40cd890 0x7efed416cf20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efec4009870 tx=0x7efec402b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2a47d45d7a6fea32 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f2655508640 1 -- 172.21.15.161:0/246799386 >> v2:172.21.15.161:3300/0 conn(0x7f26501607d0 msgr2=0x7f265015fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f2655508640 1 --2- 172.21.15.161:0/246799386 >> v2:172.21.15.161:3300/0 conn(0x7f26501607d0 0x7f265015fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f2655508640 1 -- 172.21.15.161:0/246799386 >> v1:172.21.15.161:6789/0 conn(0x7f265006cbf0 legacy=0x7f2650071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f2655508640 1 -- 172.21.15.161:0/246799386 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26500cf210 con 0x7f2650145080 2024-03-23T14:35:18.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f932fc50640 1 Processor -- start 2024-03-23T14:35:18.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f145e9d7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1450009c30 con 0x7f1450036350 2024-03-23T14:35:18.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f145cf74640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f1450038ea0 0x7f14500e2980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33904/0 (socket says 172.21.15.161:33904) 2024-03-23T14:35:18.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f145cf74640 1 -- 172.21.15.161:0/1989455512 learned_addr learned my addr 172.21.15.161:0/1989455512 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f145e9d7640 1 -- 172.21.15.161:0/1989455512 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14500068d0 con 0x7f1450038ea0 2024-03-23T14:35:18.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f145e9d7640 1 -- 172.21.15.161:0/1989455512 --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1450006bb0 con 0x7f1450035f70 2024-03-23T14:35:18.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1457fff640 1 --2- 172.21.15.161:0/1989455512 >> v2:172.21.15.161:3300/0 conn(0x7f1450035f70 0x7f1450057630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3107160029 0 0) 0x7f14500068d0 con 0x7f1450038ea0 2024-03-23T14:35:18.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1448003520 con 0x7f1450038ea0 2024-03-23T14:35:18.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa97095e640 1 -- 172.21.15.161:0/302124871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0cf5c0 msgr2=0x7fa96c0b3560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa97095e640 1 --2- 172.21.15.161:0/302124871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0cf5c0 0x7fa96c0b3560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa97095e640 1 -- 172.21.15.161:0/302124871 >> v1:172.21.15.161:6789/0 conn(0x7fa96c0b3aa0 legacy=0x7fa96c070e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fa97095e640 1 -- 172.21.15.161:0/302124871 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96c06d520 con 0x7fa96c15da90 2024-03-23T14:35:18.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.576+0000 7fe0ea90a640 1 -- start start 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7fe0ea90a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe0dc04b3c0 0x7fe0dc034020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0ea90a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc006a50 0x7fe0dc006ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0ea90a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0dc0062e0 con 0x7fe0dc04f4d0 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0ea90a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe0dc054c10 con 0x7fe0dc04b3c0 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0ea90a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe0dc053a50 con 0x7fe0dc006a50 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0ea109640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc006a50 0x7fe0dc006ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0e9908640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe0dc04b3c0 0x7fe0dc034020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0ea109640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc006a50 0x7fe0dc006ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42338/0 (socket says 172.21.15.161:42338) 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0e9908640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe0dc04b3c0 0x7fe0dc034020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41228/0 (socket says 172.21.15.161:41228) 2024-03-23T14:35:18.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0e9908640 1 -- 172.21.15.161:0/1078957904 learned_addr learned my addr 172.21.15.161:0/1078957904 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7efed159a640 1 -- 172.21.15.161:0/3272758786 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec4031870 con 0x7efed40cd890 2024-03-23T14:35:18.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7efed159a640 1 -- 172.21.15.161:0/3272758786 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec40319d0 con 0x7efed40cd890 2024-03-23T14:35:18.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fa97095e640 1 --2- 172.21.15.161:0/302124871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 0x7fa96c0b4900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009870 tx=0x7fa95c02dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e13ab19c241798 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7efed159a640 1 -- 172.21.15.161:0/3272758786 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec4031d60 con 0x7efed40cd890 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 383907369 0 0) 0x7f1448003520 con 0x7f1450038ea0 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 >> v2:172.21.15.161:3300/0 conn(0x7f1450035f70 msgr2=0x7f1450057630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 --2- 172.21.15.161:0/1989455512 >> v2:172.21.15.161:3300/0 conn(0x7f1450035f70 0x7f1450057630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 >> v1:172.21.15.161:6789/0 conn(0x7f1450036350 legacy=0x7f1450038790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1450007db0 con 0x7f1450038ea0 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f145804fcc0 con 0x7f1450038ea0 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 >> v1:172.21.15.183:6789/0 conn(0x7f1450038ea0 legacy=0x7f14500e2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448003970 0x7f1448003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448004310 0x7f14480047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1448005080 0x7f144800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0e9908640 1 -- 172.21.15.161:0/1078957904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc006a50 msgr2=0x7fe0dc006ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7efed8cae640 1 -- 172.21.15.161:0/3272758786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40cd890 msgr2=0x7efed416cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7efed8cae640 1 --2- 172.21.15.161:0/3272758786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40cd890 0x7efed416cf20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7efec4009870 tx=0x7efec402b910 comp rx=0 tx=0).stop 2024-03-23T14:35:18.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1450006bb0 con 0x7f1448003970 2024-03-23T14:35:18.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f144800bfd0 con 0x7f1448004310 2024-03-23T14:35:18.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f144800c190 con 0x7f1448005080 2024-03-23T14:35:18.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f2655508640 1 --2- 172.21.15.161:0/246799386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 0x7f2650072a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f265006c9a0 tx=0x7f264002b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=90341e66daa1de3c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f145cf74640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1448005080 0x7f144800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0e9908640 1 --2- 172.21.15.161:0/1078957904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc006a50 0x7fe0dc006ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0e9908640 1 -- 172.21.15.161:0/1078957904 >> v1:172.21.15.183:6789/0 conn(0x7fe0dc04f4d0 legacy=0x7fe0dc04f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7fe0e9908640 1 -- 172.21.15.161:0/1078957904 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0dc053d30 con 0x7fe0dc04b3c0 2024-03-23T14:35:18.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.577+0000 7f932fc50640 1 -- start start 2024-03-23T14:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.578+0000 7f14577fe640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448004310 0x7f14480047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7efed8cae640 1 -- 172.21.15.161:0/3272758786 shutdown_connections 2024-03-23T14:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7efed8cae640 1 --2- 172.21.15.161:0/3272758786 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40cd890 0x7efed416cf20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7efed8cae640 1 -- 172.21.15.161:0/3272758786 >> 172.21.15.161:0/3272758786 conn(0x7efed40d1b70 msgr2=0x7efed40d1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7fa96b7fe640 1 -- 172.21.15.161:0/302124871 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c02ea40 con 0x7fa96c15da90 2024-03-23T14:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f0d1e899640 1 Processor -- start 2024-03-23T14:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7efed8cae640 1 -- 172.21.15.161:0/3272758786 shutdown_connections 2024-03-23T14:35:18.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7efed8cae640 1 -- 172.21.15.161:0/3272758786 wait complete. 2024-03-23T14:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f145cf74640 1 -- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448004310 msgr2=0x7f14480047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f145cf74640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1448004310 0x7f14480047e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f145cf74640 1 -- 172.21.15.161:0/1989455512 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448003970 msgr2=0x7f1448003d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-23T14:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f145cf74640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448003970 0x7f1448003d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f145cf74640 1 -- 172.21.15.161:0/1989455512 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1448003520 con 0x7f1448005080 2024-03-23T14:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7fa96b7fe640 1 -- 172.21.15.161:0/302124871 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa95c02eba0 con 0x7fa96c15da90 2024-03-23T14:35:18.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7fc27a59c640 1 --2- 172.21.15.161:0/4123644765 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc274038070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc26c0097a0 tx=0x7fc26c02b640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84fab8e3bd183035 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f0d1e899640 1 -- start start 2024-03-23T14:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f932fc50640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9328161c20 0x7f93280d05a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f932fc50640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93280d0f70 con 0x7f93280d0ae0 2024-03-23T14:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f932fc50640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93280d2200 con 0x7f932815e030 2024-03-23T14:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f932fc50640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f93280cec90 con 0x7f9328161c20 2024-03-23T14:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f932d9ec640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f932815e030 0x7f93280ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33918/0 (socket says 172.21.15.161:33918) 2024-03-23T14:35:18.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f932d9ec640 1 -- 172.21.15.161:0/1139543378 learned_addr learned my addr 172.21.15.161:0/1139543378 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f2647fff640 1 -- 172.21.15.161:0/246799386 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264002e070 con 0x7f2650145080 2024-03-23T14:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f2647fff640 1 -- 172.21.15.161:0/246799386 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26400315a0 con 0x7f2650145080 2024-03-23T14:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f2647fff640 1 -- 172.21.15.161:0/246799386 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2640031930 con 0x7f2650145080 2024-03-23T14:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7fc27959a640 1 -- 172.21.15.161:0/4123644765 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c030710 con 0x7fc27400c090 2024-03-23T14:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7fc27959a640 1 -- 172.21.15.161:0/4123644765 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc26c02e050 con 0x7fc27400c090 2024-03-23T14:35:18.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7fc27959a640 1 -- 172.21.15.161:0/4123644765 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc26c030a90 con 0x7fc27400c090 2024-03-23T14:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.579+0000 7f932d1eb640 1 --2- 172.21.15.161:0/1139543378 >> v2:172.21.15.161:3300/0 conn(0x7f9328161c20 0x7f93280d05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f145cf74640 1 -- 172.21.15.161:0/1989455512 reap_dead start 2024-03-23T14:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f145cf74640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1448005080 0x7f144800b980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f145807acc0 tx=0x7f14480057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=523d56dd292f8dd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f932c9ea640 1 -- 172.21.15.161:0/1139543378 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1550097805 0 0) 0x7f93280d0f70 con 0x7f93280d0ae0 2024-03-23T14:35:18.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f932c9ea640 1 -- 172.21.15.161:0/1139543378 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9314003520 con 0x7f93280d0ae0 2024-03-23T14:35:18.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7fe0e9908640 1 --2- 172.21.15.161:0/1078957904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04b3c0 0x7fe0dc034020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009870 tx=0x7fe0e002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4578a86ed34eccb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14580af440 con 0x7f1448005080 2024-03-23T14:35:18.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f14580af5a0 con 0x7f1448005080 2024-03-23T14:35:18.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f1456ffd640 1 -- 172.21.15.161:0/1989455512 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14580af970 con 0x7f1448005080 2024-03-23T14:35:18.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f0d1e899640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d18144160 0x7f0d180b2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f0d1e899640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0d1806e520 0x7f0d18071cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f0d1e899640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d18070c30 con 0x7f0d1815f140 2024-03-23T14:35:18.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f0d1e899640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0d180718b0 con 0x7f0d1806e520 2024-03-23T14:35:18.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f0d1e899640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0d181449f0 con 0x7f0d18144160 2024-03-23T14:35:18.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7fe0e8906640 1 -- 172.21.15.161:0/1078957904 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0031640 con 0x7fe0dc04b3c0 2024-03-23T14:35:18.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f932d1eb640 1 -- 172.21.15.161:0/1139543378 >> v1:172.21.15.183:6789/0 conn(0x7f932815e030 legacy=0x7f93280ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f932d1eb640 1 -- 172.21.15.161:0/1139543378 >> v1:172.21.15.161:6789/0 conn(0x7f93280d0ae0 legacy=0x7f93280d29e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f932d1eb640 1 -- 172.21.15.161:0/1139543378 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93280cee80 con 0x7f9328161c20 2024-03-23T14:35:18.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7fa96b7fe640 1 -- 172.21.15.161:0/302124871 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa95c036440 con 0x7fa96c15da90 2024-03-23T14:35:18.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7fa972bc2640 1 -- 172.21.15.161:0/302124871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 msgr2=0x7fa96c0b4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7fa972bc2640 1 --2- 172.21.15.161:0/302124871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 0x7fa96c0b4900 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa95c009870 tx=0x7fa95c02dc90 comp rx=0 tx=0).stop 2024-03-23T14:35:18.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f265776c640 1 -- 172.21.15.161:0/246799386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 msgr2=0x7f2650072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.580+0000 7f265776c640 1 --2- 172.21.15.161:0/246799386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 0x7f2650072a00 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f265006c9a0 tx=0x7f264002b840 comp rx=0 tx=0).stop 2024-03-23T14:35:18.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7efed8cae640 1 Processor -- start 2024-03-23T14:35:18.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f265776c640 1 -- 172.21.15.161:0/246799386 shutdown_connections 2024-03-23T14:35:18.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f265776c640 1 --2- 172.21.15.161:0/246799386 >> v2:172.21.15.161:3300/0 conn(0x7f26501607d0 0x7f265015fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f265776c640 1 --2- 172.21.15.161:0/246799386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 0x7f2650072a00 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f265776c640 1 -- 172.21.15.161:0/246799386 >> 172.21.15.161:0/246799386 conn(0x7f26500b17f0 msgr2=0x7f26500b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fc280dc7640 1 -- 172.21.15.161:0/4123644765 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 msgr2=0x7fc274038070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fc280dc7640 1 --2- 172.21.15.161:0/4123644765 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc274038070 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc26c0097a0 tx=0x7fc26c02b640 comp rx=0 tx=0).stop 2024-03-23T14:35:18.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f0d1e098640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0d1806e520 0x7f0d18071cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fe0e8906640 1 -- 172.21.15.161:0/1078957904 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0e00317a0 con 0x7fe0dc04b3c0 2024-03-23T14:35:18.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f0d1d096640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0d1815f140 0x7f0d1806de10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49922/0 (socket says 172.21.15.161:49922) 2024-03-23T14:35:18.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f0d1d096640 1 -- 172.21.15.161:0/2259535408 learned_addr learned my addr 172.21.15.161:0/2259535408 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:18.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7efed8cae640 1 -- start start 2024-03-23T14:35:18.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f265776c640 1 -- 172.21.15.161:0/246799386 shutdown_connections 2024-03-23T14:35:18.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fc280dc7640 1 -- 172.21.15.161:0/4123644765 shutdown_connections 2024-03-23T14:35:18.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fc280dc7640 1 --2- 172.21.15.161:0/4123644765 >> v2:172.21.15.161:3300/0 conn(0x7fc274053c60 0x7fc2740df210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fc280dc7640 1 --2- 172.21.15.161:0/4123644765 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc274038070 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fc280dc7640 1 -- 172.21.15.161:0/4123644765 >> 172.21.15.161:0/4123644765 conn(0x7fc274009fa0 msgr2=0x7fc27400a380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fe0e8906640 1 -- 172.21.15.161:0/1078957904 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0031b30 con 0x7fe0dc04b3c0 2024-03-23T14:35:18.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f145e9d7640 1 -- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1448005080 msgr2=0x7f144800b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f145e9d7640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1448005080 0x7f144800b980 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f145807acc0 tx=0x7f14480057f0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fc280dc7640 1 -- 172.21.15.161:0/4123644765 shutdown_connections 2024-03-23T14:35:18.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7fc280dc7640 1 -- 172.21.15.161:0/4123644765 wait complete. 2024-03-23T14:35:18.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f0d1d897640 1 --2- 172.21.15.161:0/2259535408 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d18144160 0x7f0d180b2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fa972bc2640 1 -- 172.21.15.161:0/302124871 shutdown_connections 2024-03-23T14:35:18.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fa972bc2640 1 --2- 172.21.15.161:0/302124871 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0cf5c0 0x7fa96c0b3560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fa972bc2640 1 --2- 172.21.15.161:0/302124871 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 0x7fa96c0b4900 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fa972bc2640 1 -- 172.21.15.161:0/302124871 >> 172.21.15.161:0/302124871 conn(0x7fa96c0b17f0 msgr2=0x7fa96c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.605 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fe0ea90a640 1 -- 172.21.15.161:0/1078957904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04b3c0 msgr2=0x7fe0dc034020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fe0ea90a640 1 --2- 172.21.15.161:0/1078957904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04b3c0 0x7fe0dc034020 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe0e0009870 tx=0x7fe0e002b8f0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f932d1eb640 1 --2- 172.21.15.161:0/1139543378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9328161c20 0x7f93280d05a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9324009720 tx=0x7f932402b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9af764a1e4046754 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fe0ea90a640 1 -- 172.21.15.161:0/1078957904 shutdown_connections 2024-03-23T14:35:18.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fe0ea90a640 1 --2- 172.21.15.161:0/1078957904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc006a50 0x7fe0dc006ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed8cae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efed40c4b40 0x7efed40c7960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fe0ea90a640 1 --2- 172.21.15.161:0/1078957904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04b3c0 0x7fe0dc034020 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fe0ea90a640 1 -- 172.21.15.161:0/1078957904 >> 172.21.15.161:0/1078957904 conn(0x7fe0dc00a400 msgr2=0x7fe0dc00a7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed8cae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed40cd890 0x7efed40c6fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed8cae640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40d2690 0x7efed4159000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed8cae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efed40c5920 con 0x7efed40cd890 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed2d9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40d2690 0x7efed4159000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed259c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efed40c4b40 0x7efed40c7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed2d9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40d2690 0x7efed4159000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:38930/0 (socket says 172.21.15.161:38930) 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed2d9d640 1 -- 172.21.15.161:0/1348782869 learned_addr learned my addr 172.21.15.161:0/1348782869 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efed40c8500 con 0x7efed40d2690 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed1d9b640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed40cd890 0x7efed40c6fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efed40c55f0 con 0x7efed40c4b40 2024-03-23T14:35:18.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f932c9ea640 1 -- 172.21.15.161:0/1139543378 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9324030710 con 0x7f9328161c20 2024-03-23T14:35:18.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f932c9ea640 1 -- 172.21.15.161:0/1139543378 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f932402e050 con 0x7f9328161c20 2024-03-23T14:35:18.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f932c9ea640 1 -- 172.21.15.161:0/1139543378 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9324030a90 con 0x7f9328161c20 2024-03-23T14:35:18.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f265776c640 1 -- 172.21.15.161:0/246799386 wait complete. 2024-03-23T14:35:18.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fa972bc2640 1 -- 172.21.15.161:0/302124871 shutdown_connections 2024-03-23T14:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7fa972bc2640 1 -- 172.21.15.161:0/302124871 wait complete. 2024-03-23T14:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f0d1e098640 1 -- 172.21.15.161:0/2259535408 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d18144160 msgr2=0x7f0d180b2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f0d1e098640 1 --2- 172.21.15.161:0/2259535408 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d18144160 0x7f0d180b2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f0d1e098640 1 -- 172.21.15.161:0/2259535408 >> v1:172.21.15.161:6789/0 conn(0x7f0d1815f140 legacy=0x7f0d1806de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.610 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.581+0000 7f0d1e098640 1 -- 172.21.15.161:0/2259535408 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d18070f10 con 0x7f0d1806e520 2024-03-23T14:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7f265776c640 1 Processor -- start 2024-03-23T14:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7f265776c640 1 -- start start 2024-03-23T14:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7f0d1e098640 1 --2- 172.21.15.161:0/2259535408 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d1806e520 0x7f0d18071cd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000b080 tx=0x7f0d1002c120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81fd8f40d39bbb06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7fe0ea90a640 1 -- 172.21.15.161:0/1078957904 shutdown_connections 2024-03-23T14:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7efed1d9b640 1 -- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efed40c4b40 msgr2=0x7efed40c7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7efed1d9b640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efed40c4b40 0x7efed40c7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7efed1d9b640 1 -- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40d2690 msgr2=0x7efed4159000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7efed1d9b640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40d2690 0x7efed4159000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7fe0ea90a640 1 -- 172.21.15.161:0/1078957904 wait complete. 2024-03-23T14:35:18.612 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7f0d1c895640 1 -- 172.21.15.161:0/2259535408 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1002ccc0 con 0x7f0d1806e520 2024-03-23T14:35:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7f0d1c895640 1 -- 172.21.15.161:0/2259535408 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d1002ce20 con 0x7f0d1806e520 2024-03-23T14:35:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f145e9d7640 1 -- 172.21.15.161:0/1989455512 shutdown_connections 2024-03-23T14:35:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f145e9d7640 1 --2- 172.21.15.161:0/1989455512 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1448005080 0x7f144800b980 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f145e9d7640 1 -- 172.21.15.161:0/1989455512 >> 172.21.15.161:0/1989455512 conn(0x7f14500200a0 msgr2=0x7f1450056d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f145e9d7640 1 -- 172.21.15.161:0/1989455512 shutdown_connections 2024-03-23T14:35:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.582+0000 7f145e9d7640 1 -- 172.21.15.161:0/1989455512 wait complete. 2024-03-23T14:35:18.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7fc280dc7640 1 Processor -- start 2024-03-23T14:35:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7efed1d9b640 1 -- 172.21.15.161:0/1348782869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec40094d0 con 0x7efed40cd890 2024-03-23T14:35:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f265776c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f265006cbf0 0x7f2650157610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7efed1d9b640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed40cd890 0x7efed40c6fa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efec800c410 tx=0x7efec800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f265776c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 0x7f2650159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f265776c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26501607d0 0x7f2650159760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f265776c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2650159ca0 con 0x7f26501607d0 2024-03-23T14:35:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f265776c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f265015b1a0 con 0x7f2650145080 2024-03-23T14:35:18.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f265776c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f265015b440 con 0x7f265006cbf0 2024-03-23T14:35:18.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f932fc50640 1 -- 172.21.15.161:0/1139543378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9328161c20 msgr2=0x7f93280d05a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f932fc50640 1 --2- 172.21.15.161:0/1139543378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9328161c20 0x7f93280d05a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9324009720 tx=0x7f932402b730 comp rx=0 tx=0).stop 2024-03-23T14:35:18.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f145e9d7640 1 Processor -- start 2024-03-23T14:35:18.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f145e9d7640 1 -- start start 2024-03-23T14:35:18.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f2655508640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f265006cbf0 0x7f2650157610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f2655508640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f265006cbf0 0x7f2650157610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42370/0 (socket says 172.21.15.161:42370) 2024-03-23T14:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f2655d09640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26501607d0 0x7f2650159760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f2655508640 1 -- 172.21.15.161:0/2142312216 learned_addr learned my addr 172.21.15.161:0/2142312216 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7fa972bc2640 1 Processor -- start 2024-03-23T14:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7fa972bc2640 1 -- start start 2024-03-23T14:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7efec2ffd640 1 -- 172.21.15.161:0/1348782869 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec80035f0 con 0x7efed40cd890 2024-03-23T14:35:18.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7efec2ffd640 1 -- 172.21.15.161:0/1348782869 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efec8003770 con 0x7efed40cd890 2024-03-23T14:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7efec2ffd640 1 -- 172.21.15.161:0/1348782869 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efec8003b60 con 0x7efed40cd890 2024-03-23T14:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f932fc50640 1 -- 172.21.15.161:0/1139543378 shutdown_connections 2024-03-23T14:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f932fc50640 1 --2- 172.21.15.161:0/1139543378 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9328161c20 0x7f93280d05a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f932fc50640 1 -- 172.21.15.161:0/1139543378 >> 172.21.15.161:0/1139543378 conn(0x7f93280b5a10 msgr2=0x7f93280b5e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efed40c7f40 con 0x7efed40cd890 2024-03-23T14:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f0d1e899640 1 -- 172.21.15.161:0/2259535408 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d1806e520 msgr2=0x7f0d18071cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f0d1e899640 1 --2- 172.21.15.161:0/2259535408 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d1806e520 0x7f0d18071cd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000b080 tx=0x7f0d1002c120 comp rx=0 tx=0).stop 2024-03-23T14:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f0d1e899640 1 -- 172.21.15.161:0/2259535408 shutdown_connections 2024-03-23T14:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f0d1e899640 1 --2- 172.21.15.161:0/2259535408 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d1806e520 0x7f0d18071cd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f0d1e899640 1 --2- 172.21.15.161:0/2259535408 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d18144160 0x7f0d180b2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7f0d1e899640 1 -- 172.21.15.161:0/2259535408 >> 172.21.15.161:0/2259535408 conn(0x7f0d180b17f0 msgr2=0x7f0d180b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efed40c74e0 con 0x7efed40cd890 2024-03-23T14:35:18.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.583+0000 7fc280dc7640 1 -- start start 2024-03-23T14:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.584+0000 7fc280dc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc2740485e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc280dc7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc274033ee0 0x7fc274047110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc280dc7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc274053c60 0x7fc2740cacd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc280dc7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc27404afc0 con 0x7fc274053c60 2024-03-23T14:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc280dc7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc27404a9e0 con 0x7fc274033ee0 2024-03-23T14:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc280dc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc2740cb210 con 0x7fc27400c090 2024-03-23T14:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f2654d07640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 0x7f2650159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f0d1e899640 1 -- 172.21.15.161:0/2259535408 shutdown_connections 2024-03-23T14:35:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f2654d07640 1 -- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f265006cbf0 msgr2=0x7f2650157610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f2654d07640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f265006cbf0 0x7f2650157610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f2654d07640 1 -- 172.21.15.161:0/2142312216 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26501607d0 msgr2=0x7f2650159760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f145e9d7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448005080 0x7f1450166dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f145e9d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14500370b0 0x7f1450180610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc274053c60 0x7fc2740cacd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc274053c60 0x7fc2740cacd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41280/0 (socket says 172.21.15.161:41280) 2024-03-23T14:35:18.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27ad9d640 1 -- 172.21.15.161:0/495734793 learned_addr learned my addr 172.21.15.161:0/495734793 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27a59c640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc2740485e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27ad9d640 1 -- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 msgr2=0x7fc2740485e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27ad9d640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc2740485e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27ad9d640 1 -- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc274033ee0 msgr2=0x7fc274047110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f145e9d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f145016c900 0x7f145016cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f145e9d7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f14500381c0 con 0x7f1448005080 2024-03-23T14:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f145e9d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f14500375b0 con 0x7f145016c900 2024-03-23T14:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f145e9d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1450037890 con 0x7f14500370b0 2024-03-23T14:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f14577fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14500370b0 0x7f1450180610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fa972bc2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0b3aa0 0x7fa96c1f8f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f14577fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14500370b0 0x7f1450180610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42388/0 (socket says 172.21.15.161:42388) 2024-03-23T14:35:18.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f14577fe640 1 -- 172.21.15.161:0/1296678253 learned_addr learned my addr 172.21.15.161:0/1296678253 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f932fc50640 1 -- 172.21.15.161:0/1139543378 shutdown_connections 2024-03-23T14:35:18.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f2654d07640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26501607d0 0x7f2650159760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f2654d07640 1 -- 172.21.15.161:0/2142312216 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2640009400 con 0x7f2650145080 2024-03-23T14:35:18.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27ad9d640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc274033ee0 0x7fc274047110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27ad9d640 1 -- 172.21.15.161:0/495734793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc26c009400 con 0x7fc274053c60 2024-03-23T14:35:18.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7fc27a59c640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc2740485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:35:18.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fc27ad9d640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc274053c60 0x7fc2740cacd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc27c06ab50 tx=0x7fc27c02d390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa972bc2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa96c0cf5c0 0x7fa96c148a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f2655d09640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26501607d0 0x7f2650159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:18.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa97095e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0b3aa0 0x7fa96c1f8f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa97095e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0b3aa0 0x7fa96c1f8f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42400/0 (socket says 172.21.15.161:42400) 2024-03-23T14:35:18.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f145cf74640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f145016c900 0x7f145016cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f1457fff640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448005080 0x7f1450166dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f14577fe640 1 -- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f145016c900 msgr2=0x7f145016cd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f14577fe640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f145016c900 0x7f145016cd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f14577fe640 1 -- 172.21.15.161:0/1296678253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448005080 msgr2=0x7f1450166dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa97095e640 1 -- 172.21.15.161:0/4260343870 learned_addr learned my addr 172.21.15.161:0/4260343870 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f2654d07640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 0x7f2650159220 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f264c00b3e0 tx=0x7f264c00b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa96bfff640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa96c0cf5c0 0x7fa96c148a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fc26affd640 1 -- 172.21.15.161:0/495734793 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c073480 con 0x7fc274053c60 2024-03-23T14:35:18.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa972bc2640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 0x7fa96c1597e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f2645ffb640 1 -- 172.21.15.161:0/2142312216 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c011830 con 0x7f2650145080 2024-03-23T14:35:18.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c159ff0 con 0x7fa96c15da90 2024-03-23T14:35:18.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa96c152960 con 0x7fa96c0cf5c0 2024-03-23T14:35:18.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa96c15b3e0 con 0x7fa96c0b3aa0 2024-03-23T14:35:18.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2740de6c0 con 0x7fc274053c60 2024-03-23T14:35:18.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc27404b450 con 0x7fc274053c60 2024-03-23T14:35:18.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fc26affd640 1 -- 172.21.15.161:0/495734793 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc27c02d580 con 0x7fc274053c60 2024-03-23T14:35:18.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fc26affd640 1 -- 172.21.15.161:0/495734793 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc27c071750 con 0x7fc274053c60 2024-03-23T14:35:18.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f265015b720 con 0x7f2650145080 2024-03-23T14:35:18.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f2645ffb640 1 -- 172.21.15.161:0/2142312216 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f264c011990 con 0x7f2650145080 2024-03-23T14:35:18.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f2645ffb640 1 -- 172.21.15.161:0/2142312216 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f264c011d40 con 0x7f2650145080 2024-03-23T14:35:18.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.587+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f265015bce0 con 0x7f2650145080 2024-03-23T14:35:18.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.587+0000 7fe0ea90a640 1 Processor -- start 2024-03-23T14:35:18.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7f932fc50640 1 -- 172.21.15.161:0/1139543378 wait complete. 2024-03-23T14:35:18.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f932fc50640 1 Processor -- start 2024-03-23T14:35:18.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7fe0ea90a640 1 -- start start 2024-03-23T14:35:18.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.585+0000 7f0d1e899640 1 -- 172.21.15.161:0/2259535408 wait complete. 2024-03-23T14:35:18.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc260005290 con 0x7fc274053c60 2024-03-23T14:35:18.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.587+0000 7f0d1e899640 1 Processor -- start 2024-03-23T14:35:18.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7efec2ffd640 1 -- 172.21.15.161:0/1348782869 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efec8003cc0 con 0x7efed40cd890 2024-03-23T14:35:18.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7efec2ffd640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb406b8e0 0x7efeb406dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f14577fe640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448005080 0x7f1450166dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f14577fe640 1 -- 172.21.15.161:0/1296678253 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1450037b70 con 0x7f14500370b0 2024-03-23T14:35:18.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7efec2ffd640 1 -- 172.21.15.161:0/1348782869 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efec80895f0 con 0x7efed40cd890 2024-03-23T14:35:18.633 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f932fc50640 1 -- start start 2024-03-23T14:35:18.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f1457fff640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448005080 0x7f1450166dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:35:18.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f14577fe640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14500370b0 0x7f1450180610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f144c00d4f0 tx=0x7f144c00d9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7fc26affd640 1 -- 172.21.15.161:0/495734793 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc27c0718b0 con 0x7fc274053c60 2024-03-23T14:35:18.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f2645ffb640 1 -- 172.21.15.161:0/2142312216 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f264c01a400 con 0x7f2650145080 2024-03-23T14:35:18.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f265015bf00 con 0x7f2650145080 2024-03-23T14:35:18.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f1454ff9640 1 -- 172.21.15.161:0/1296678253 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c003610 con 0x7f14500370b0 2024-03-23T14:35:18.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f1454ff9640 1 -- 172.21.15.161:0/1296678253 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f144c003770 con 0x7f14500370b0 2024-03-23T14:35:18.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f932fc50640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93280d0ae0 0x7f9328201470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7fc26affd640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc25c06b840 0x7fc25c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f932fc50640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f932815e030 0x7f9328160f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7fc26affd640 1 -- 172.21.15.161:0/495734793 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc27c104ef0 con 0x7fc274053c60 2024-03-23T14:35:18.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f2645ffb640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f263406b860 0x7f263406dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7fc27a59c640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc25c06b840 0x7fc25c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f1454ff9640 1 -- 172.21.15.161:0/1296678253 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144c003ae0 con 0x7f14500370b0 2024-03-23T14:35:18.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.586+0000 7fa97115f640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 0x7fa96c1597e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7fa97115f640 1 -- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0b3aa0 msgr2=0x7fa96c1f8f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7fa97115f640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0b3aa0 0x7fa96c1f8f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7fa97115f640 1 -- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa96c0cf5c0 msgr2=0x7fa96c148a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7fa97115f640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa96c0cf5c0 0x7fa96c148a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7fa97115f640 1 -- 172.21.15.161:0/4260343870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa95c0094d0 con 0x7fa96c15da90 2024-03-23T14:35:18.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f145e9d7640 1 -- 172.21.15.161:0/1296678253 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1450037e50 con 0x7f14500370b0 2024-03-23T14:35:18.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f145e9d7640 1 -- 172.21.15.161:0/1296678253 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f1450167310 con 0x7f14500370b0 2024-03-23T14:35:18.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f2645ffb640 1 -- 172.21.15.161:0/2142312216 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f264c08a060 con 0x7f2650145080 2024-03-23T14:35:18.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7fe0ea90a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0dc006a50 0x7fe0dc0eb6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f2655508640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f263406b860 0x7f263406dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7fa97115f640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 0x7fa96c1597e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa96400c780 tx=0x7fa96400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.587+0000 7f0d1e899640 1 -- start start 2024-03-23T14:35:18.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e899640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d1806e520 0x7f0d1815d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e899640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d18144160 0x7f0d18159240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e899640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1815f140 0x7f0d18159780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e899640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0d180d0720 con 0x7f0d18144160 2024-03-23T14:35:18.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e899640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0d18158cd0 con 0x7f0d1806e520 2024-03-23T14:35:18.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e899640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1815c840 con 0x7f0d1815f140 2024-03-23T14:35:18.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e098640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1815f140 0x7f0d18159780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e098640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1815f140 0x7f0d18159780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:42408/0 (socket says 172.21.15.161:42408) 2024-03-23T14:35:18.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.588+0000 7f0d1e098640 1 -- 172.21.15.161:0/895227175 learned_addr learned my addr 172.21.15.161:0/895227175 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.589+0000 7f932d9ec640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93280d0ae0 0x7f9328201470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f932d9ec640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93280d0ae0 0x7f9328201470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:38984/0 (socket says 172.21.15.161:38984) 2024-03-23T14:35:18.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f932d9ec640 1 -- 172.21.15.161:0/3721274175 learned_addr learned my addr 172.21.15.161:0/3721274175 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f0d1d897640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d1806e520 0x7f0d1815d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f932d1eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f932815e030 0x7f9328160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7fc27a59c640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc25c06b840 0x7fc25c06dd00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc26c034000 tx=0x7fc26c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f932fc50640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9328161c20 0x7f93281fa860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7fe0ea90a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc04b3c0 0x7fe0dc0ca4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f932fc50640 1 -- 172.21.15.161:0/3721274175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f932806daa0 con 0x7f932815e030 2024-03-23T14:35:18.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7efed259c640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb406b8e0 0x7efeb406dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f0d1d096640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d18144160 0x7f0d18159240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f932e1ed640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9328161c20 0x7f93281fa860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f932fc50640 1 -- 172.21.15.161:0/3721274175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f932806d290 con 0x7f93280d0ae0 2024-03-23T14:35:18.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7fe0e9908640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0dc006a50 0x7fe0dc0eb6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7fe0ea90a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04f4d0 0x7fe0dc035420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f932fc50640 1 -- 172.21.15.161:0/3721274175 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9328161460 con 0x7f9328161c20 2024-03-23T14:35:18.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7fe0e9908640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0dc006a50 0x7fe0dc0eb6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:39012/0 (socket says 172.21.15.161:39012) 2024-03-23T14:35:18.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f2655508640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f263406b860 0x7f263406dd20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f264002b880 tx=0x7f264003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efeb8005290 con 0x7efed40cd890 2024-03-23T14:35:18.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7fe0ea109640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04f4d0 0x7fe0dc035420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f0d1e098640 1 -- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d1806e520 msgr2=0x7f0d1815d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f0d1e098640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d1806e520 0x7f0d1815d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f0d1e098640 1 -- 172.21.15.161:0/895227175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d18144160 msgr2=0x7f0d18159240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f0d1e098640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d18144160 0x7f0d18159240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.590+0000 7f0d1e098640 1 -- 172.21.15.161:0/895227175 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d10009a70 con 0x7f0d1815f140 2024-03-23T14:35:18.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0ea109640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04f4d0 0x7fe0dc035420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41318/0 (socket says 172.21.15.161:41318) 2024-03-23T14:35:18.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0e9908640 1 -- 172.21.15.161:0/232622351 learned_addr learned my addr 172.21.15.161:0/232622351 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:18.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f0d1e098640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1815f140 0x7f0d18159780 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000b1a0 tx=0x7f0d10004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0ea90a640 1 -- 172.21.15.161:0/232622351 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe0dc04cb60 con 0x7fe0dc04f4d0 2024-03-23T14:35:18.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0ea90a640 1 -- 172.21.15.161:0/232622351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe0dc0ca240 con 0x7fe0dc006a50 2024-03-23T14:35:18.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0ea90a640 1 -- 172.21.15.161:0/232622351 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe0dc04bf50 con 0x7fe0dc04b3c0 2024-03-23T14:35:18.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f145e9d7640 1 -- 172.21.15.161:0/1296678253 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1450038370 con 0x7f14500370b0 2024-03-23T14:35:18.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f932d9ec640 1 -- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9328161c20 msgr2=0x7f93281fa860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f932d9ec640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9328161c20 0x7f93281fa860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f932d9ec640 1 -- 172.21.15.161:0/3721274175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f932815e030 msgr2=0x7f9328160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0e9107640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc04b3c0 0x7fe0dc0ca4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f0d0e7fc640 1 -- 172.21.15.161:0/895227175 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d10034a70 con 0x7f0d1815f140 2024-03-23T14:35:18.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f0d0e7fc640 1 -- 172.21.15.161:0/895227175 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d10034bd0 con 0x7f0d1815f140 2024-03-23T14:35:18.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f932d9ec640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f932815e030 0x7f9328160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f0d0e7fc640 1 -- 172.21.15.161:0/895227175 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d1003ee60 con 0x7f0d1815f140 2024-03-23T14:35:18.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f932d9ec640 1 -- 172.21.15.161:0/3721274175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9324009400 con 0x7f93280d0ae0 2024-03-23T14:35:18.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f932d1eb640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f932815e030 0x7f9328160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:18.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d1815cb20 con 0x7f0d1815f140 2024-03-23T14:35:18.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d18158fb0 con 0x7f0d1815f140 2024-03-23T14:35:18.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7f932d9ec640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93280d0ae0 0x7f9328201470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f931800c780 tx=0x7f931800cc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7efed259c640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb406b8e0 0x7efeb406dda0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efecc009fc0 tx=0x7efecc00d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0e9908640 1 -- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc04b3c0 msgr2=0x7fe0dc0ca4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0e9908640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc04b3c0 0x7fe0dc0ca4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0e9908640 1 -- 172.21.15.161:0/232622351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04f4d0 msgr2=0x7fe0dc035420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:18.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0e9908640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04f4d0 0x7fe0dc035420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:18.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0e9908640 1 -- 172.21.15.161:0/232622351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e00094d0 con 0x7fe0dc006a50 2024-03-23T14:35:18.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.591+0000 7fe0ea109640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04f4d0 0x7fe0dc035420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:18.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.592+0000 7fe0e9908640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0dc006a50 0x7fe0dc0eb6b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00090b0 tx=0x7fe0e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.592+0000 7f1454ff9640 1 -- 172.21.15.161:0/1296678253 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f144c003c40 con 0x7f14500370b0 2024-03-23T14:35:18.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.592+0000 7f1454ff9640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f143c06b890 0x7f143c06dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.592+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d1815d040 con 0x7f0d1815f140 2024-03-23T14:35:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.592+0000 7f1454ff9640 1 -- 172.21.15.161:0/1296678253 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f144c08a070 con 0x7f14500370b0 2024-03-23T14:35:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.592+0000 7f1457fff640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f143c06b890 0x7f143c06dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.593+0000 7f1457fff640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f143c06b890 0x7f143c06dd50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1450056960 tx=0x7f1444008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.593+0000 7f0d0e7fc640 1 -- 172.21.15.161:0/895227175 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0d10046030 con 0x7f0d1815f140 2024-03-23T14:35:18.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.594+0000 7f0d0e7fc640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0cfc06b8e0 0x7f0cfc06dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.594+0000 7f0d0e7fc640 1 -- 172.21.15.161:0/895227175 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0d100b07a0 con 0x7f0d1815f140 2024-03-23T14:35:18.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.594+0000 7f0d1d897640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0cfc06b8e0 0x7f0cfc06dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.594+0000 7f0d1d897640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0cfc06b8e0 0x7f0cfc06dda0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0d14006fd0 tx=0x7f0d14008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.594+0000 7f931e7fc640 1 -- 172.21.15.161:0/3721274175 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93180072d0 con 0x7f93280d0ae0 2024-03-23T14:35:18.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7f931e7fc640 1 -- 172.21.15.161:0/3721274175 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9318007430 con 0x7f93280d0ae0 2024-03-23T14:35:18.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7f932fc50640 1 -- 172.21.15.161:0/3721274175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f93281fada0 con 0x7f93280d0ae0 2024-03-23T14:35:18.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7fe0da7fc640 1 -- 172.21.15.161:0/232622351 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e003a930 con 0x7fe0dc006a50 2024-03-23T14:35:18.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7fe0da7fc640 1 -- 172.21.15.161:0/232622351 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0e003aab0 con 0x7fe0dc006a50 2024-03-23T14:35:18.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7fe0ea90a640 1 -- 172.21.15.161:0/232622351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0dc04c230 con 0x7fe0dc006a50 2024-03-23T14:35:18.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7fe0da7fc640 1 -- 172.21.15.161:0/232622351 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e003ae60 con 0x7fe0dc006a50 2024-03-23T14:35:18.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7fe0ea90a640 1 -- 172.21.15.161:0/232622351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0dc04c7f0 con 0x7fe0dc006a50 2024-03-23T14:35:18.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7f931e7fc640 1 -- 172.21.15.161:0/3721274175 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f93180047e0 con 0x7f93280d0ae0 2024-03-23T14:35:18.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.595+0000 7f932fc50640 1 -- 172.21.15.161:0/3721274175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f932806d7c0 con 0x7f93280d0ae0 2024-03-23T14:35:18.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.596+0000 7f2645ffb640 1 -- 172.21.15.161:0/2142312216 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f264c0bde10 con 0x7f2650145080 2024-03-23T14:35:18.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.596+0000 7fe0da7fc640 1 -- 172.21.15.161:0/232622351 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe0e002f050 con 0x7fe0dc006a50 2024-03-23T14:35:18.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.596+0000 7fe0ea90a640 1 -- 172.21.15.161:0/232622351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0dc008e20 con 0x7fe0dc006a50 2024-03-23T14:35:18.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7f931e7fc640 1 -- 172.21.15.161:0/3721274175 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9318004a00 con 0x7f93280d0ae0 2024-03-23T14:35:18.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7fe0da7fc640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0c806b810 0x7fe0c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7f932fc50640 1 -- 172.21.15.161:0/3721274175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9328162000 con 0x7f93280d0ae0 2024-03-23T14:35:18.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7fe0da7fc640 1 -- 172.21.15.161:0/232622351 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe0e00ac070 con 0x7fe0dc006a50 2024-03-23T14:35:18.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7fe0e9107640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0c806b810 0x7fe0c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7fc26affd640 1 -- 172.21.15.161:0/495734793 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc27c0cf920 con 0x7fc274053c60 2024-03-23T14:35:18.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7fa9697fa640 1 -- 172.21.15.161:0/4260343870 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96400a840 con 0x7fa96c15da90 2024-03-23T14:35:18.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96c148fc0 con 0x7fa96c15da90 2024-03-23T14:35:18.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa96c152b70 con 0x7fa96c15da90 2024-03-23T14:35:18.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7f1454ff9640 1 -- 172.21.15.161:0/1296678253 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f144c0bde10 con 0x7f14500370b0 2024-03-23T14:35:18.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.597+0000 7f931e7fc640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f930c06b810 0x7f930c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.598+0000 7fe0e9107640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0c806b810 0x7fe0c806dcd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc04a280 tx=0x7fe0d4009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.598+0000 7f932d1eb640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f930c06b810 0x7f930c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.598+0000 7f932d1eb640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f930c06b810 0x7f930c06dcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9324034000 tx=0x7f9324008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:18.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.598+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa96c0735e0 con 0x7fa96c15da90 2024-03-23T14:35:18.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.601+0000 7f931e7fc640 1 -- 172.21.15.161:0/3721274175 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9318057eb0 con 0x7f93280d0ae0 2024-03-23T14:35:18.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.601+0000 7fe0da7fc640 1 -- 172.21.15.161:0/232622351 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe0e00dfe10 con 0x7fe0dc006a50 2024-03-23T14:35:18.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.603+0000 7f0d0e7fc640 1 -- 172.21.15.161:0/895227175 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0d1815d040 con 0x7f0d1815f140 2024-03-23T14:35:18.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.606+0000 7efec2ffd640 1 -- 172.21.15.161:0/1348782869 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efec8054130 con 0x7efed40cd890 2024-03-23T14:35:18.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.606+0000 7f931e7fc640 1 -- 172.21.15.161:0/3721274175 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9318054150 con 0x7f93280d0ae0 2024-03-23T14:35:18.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.606+0000 7fa9697fa640 1 -- 172.21.15.161:0/4260343870 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa964003d90 con 0x7fa96c15da90 2024-03-23T14:35:18.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.606+0000 7fa9697fa640 1 -- 172.21.15.161:0/4260343870 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96400aba0 con 0x7fa96c15da90 2024-03-23T14:35:18.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.617+0000 7fa9697fa640 1 -- 172.21.15.161:0/4260343870 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa96400ad00 con 0x7fa96c15da90 2024-03-23T14:35:18.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.618+0000 7fa9697fa640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa95806b810 0x7fa95806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:18.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.618+0000 7fa9697fa640 1 -- 172.21.15.161:0/4260343870 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa96408a940 con 0x7fa96c15da90 2024-03-23T14:35:18.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.618+0000 7fa9697fa640 1 -- 172.21.15.161:0/4260343870 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9640b7e90 con 0x7fa96c15da90 2024-03-23T14:35:18.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.619+0000 7fa97095e640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa95806b810 0x7fa95806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:18.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:18.619+0000 7fa97095e640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa95806b810 0x7fa95806dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa95c000c00 tx=0x7fa95c032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.006+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f26500735e0 con 0x7f2650145080 2024-03-23T14:35:19.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.007+0000 7f2645ffb640 1 -- 172.21.15.161:0/2142312216 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f264c015020 con 0x7f2650145080 2024-03-23T14:35:19.004 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-23T14:35:19.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.007+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fc260005540 con 0x7fc274053c60 2024-03-23T14:35:19.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.008+0000 7fc26affd640 1 -- 172.21.15.161:0/495734793 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fc27c0d2fe0 con 0x7fc274053c60 2024-03-23T14:35:19.005 INFO:teuthology.orchestra.run.smithi161.stdout:55834574854 2024-03-23T14:35:19.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.010+0000 7fe0ea90a640 1 -- 172.21.15.161:0/232622351 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fe0dc0057e0 con 0x7fe0dc006a50 2024-03-23T14:35:19.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.010+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f263406b860 msgr2=0x7f263406dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.010+0000 7f265776c640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f263406b860 0x7f263406dd20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f264002b880 tx=0x7f264003d040 comp rx=0 tx=0).stop 2024-03-23T14:35:19.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.010+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 msgr2=0x7f2650159220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.010+0000 7f265776c640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 0x7f2650159220 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f264c00b3e0 tx=0x7f264c00b8a0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 shutdown_connections 2024-03-23T14:35:19.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7fe0da7fc640 1 -- 172.21.15.161:0/232622351 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe0e0035020 con 0x7fe0dc006a50 2024-03-23T14:35:19.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7f265776c640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f263406b860 0x7f263406dd20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7f265776c640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f26501607d0 0x7f2650159760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.012 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-23T14:35:19.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7f265776c640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2650145080 0x7f2650159220 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7f265776c640 1 --2- 172.21.15.161:0/2142312216 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f265006cbf0 0x7f2650157610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 >> 172.21.15.161:0/2142312216 conn(0x7f26500b17f0 msgr2=0x7f2650140e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 shutdown_connections 2024-03-23T14:35:19.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7f265776c640 1 -- 172.21.15.161:0/2142312216 wait complete. 2024-03-23T14:35:19.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc25c06b840 msgr2=0x7fc25c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.011+0000 7fc280dc7640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc25c06b840 0x7fc25c06dd00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fc26c034000 tx=0x7fc26c008030 comp rx=0 tx=0).stop 2024-03-23T14:35:19.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc274053c60 msgr2=0x7fc2740cacd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc274053c60 0x7fc2740cacd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc27c06ab50 tx=0x7fc27c02d390 comp rx=0 tx=0).stop 2024-03-23T14:35:19.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 shutdown_connections 2024-03-23T14:35:19.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc25c06b840 0x7fc25c06dd00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc274053c60 0x7fc2740cacd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc274033ee0 0x7fc274047110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 --2- 172.21.15.161:0/495734793 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc27400c090 0x7fc2740485e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 >> 172.21.15.161:0/495734793 conn(0x7fc274009fa0 msgr2=0x7fc274034890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 shutdown_connections 2024-03-23T14:35:19.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.012+0000 7fc280dc7640 1 -- 172.21.15.161:0/495734793 wait complete. 2024-03-23T14:35:19.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 -- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0c806b810 msgr2=0x7fe0c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0c806b810 0x7fe0c806dcd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc04a280 tx=0x7fe0d4009060 comp rx=0 tx=0).stop 2024-03-23T14:35:19.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 -- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0dc006a50 msgr2=0x7fe0dc0eb6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0dc006a50 0x7fe0dc0eb6b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe0e00090b0 tx=0x7fe0e00046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 -- 172.21.15.161:0/232622351 shutdown_connections 2024-03-23T14:35:19.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0c806b810 0x7fe0c806dcd0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc04a280 tx=0x7fe0d4009060 comp rx=0 tx=0).stop 2024-03-23T14:35:19.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0dc04f4d0 0x7fe0dc035420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0dc04b3c0 0x7fe0dc0ca4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 --2- 172.21.15.161:0/232622351 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0dc006a50 0x7fe0dc0eb6b0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.014+0000 7fe0c7fff640 1 -- 172.21.15.161:0/232622351 >> 172.21.15.161:0/232622351 conn(0x7fe0dc00a400 msgr2=0x7fe0dc037270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.015+0000 7fe0c7fff640 1 -- 172.21.15.161:0/232622351 shutdown_connections 2024-03-23T14:35:19.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.015+0000 7f145e9d7640 1 -- 172.21.15.161:0/1296678253 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f1450037fe0 con 0x7f14500370b0 2024-03-23T14:35:19.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.015+0000 7fe0c7fff640 1 -- 172.21.15.161:0/232622351 wait complete. 2024-03-23T14:35:19.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.015+0000 7f1454ff9640 1 -- 172.21.15.161:0/1296678253 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f1450038370 con 0x7f14500370b0 2024-03-23T14:35:19.018 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-23T14:35:19.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.015+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7efeb8005540 con 0x7efed40cd890 2024-03-23T14:35:19.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.016+0000 7efec2ffd640 1 -- 172.21.15.161:0/1348782869 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efec8017020 con 0x7efed40cd890 2024-03-23T14:35:19.019 INFO:teuthology.orchestra.run.smithi161.stdout:55834574855 2024-03-23T14:35:19.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.019+0000 7f14427fc640 1 -- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f143c06b890 msgr2=0x7f143c06dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.019+0000 7f14427fc640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f143c06b890 0x7f143c06dd50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1450056960 tx=0x7f1444008040 comp rx=0 tx=0).stop 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.019+0000 7f14427fc640 1 -- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14500370b0 msgr2=0x7f1450180610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.019+0000 7f14427fc640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14500370b0 0x7f1450180610 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f144c00d4f0 tx=0x7f144c00d9b0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.019+0000 7f14427fc640 1 -- 172.21.15.161:0/1296678253 shutdown_connections 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.019+0000 7f14427fc640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f143c06b890 0x7f143c06dd50 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f1450056960 tx=0x7f1444008040 comp rx=0 tx=0).stop 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7f14427fc640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f145016c900 0x7f145016cd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb406b8e0 msgr2=0x7efeb406dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb406b8e0 0x7efeb406dda0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7efecc009fc0 tx=0x7efecc00d040 comp rx=0 tx=0).stop 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed40cd890 msgr2=0x7efed40c6fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed40cd890 0x7efed40c6fa0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7efec800c410 tx=0x7efec800c810 comp rx=0 tx=0).stop 2024-03-23T14:35:19.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7f14427fc640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f14500370b0 0x7f1450180610 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7f14427fc640 1 --2- 172.21.15.161:0/1296678253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1448005080 0x7f1450166dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7f14427fc640 1 -- 172.21.15.161:0/1296678253 >> 172.21.15.161:0/1296678253 conn(0x7f14500200a0 msgr2=0x7f1450057610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 shutdown_connections 2024-03-23T14:35:19.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb406b8e0 0x7efeb406dda0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efed40d2690 0x7efed4159000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efed40cd890 0x7efed40c6fa0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 --2- 172.21.15.161:0/1348782869 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efed40c4b40 0x7efed40c7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 >> 172.21.15.161:0/1348782869 conn(0x7efed40d1b70 msgr2=0x7efed415ba60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 shutdown_connections 2024-03-23T14:35:19.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.020+0000 7efed8cae640 1 -- 172.21.15.161:0/1348782869 wait complete. 2024-03-23T14:35:19.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.021+0000 7f14427fc640 1 -- 172.21.15.161:0/1296678253 shutdown_connections 2024-03-23T14:35:19.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.021+0000 7f14427fc640 1 -- 172.21.15.161:0/1296678253 wait complete. 2024-03-23T14:35:19.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.022+0000 7f932fc50640 1 -- 172.21.15.161:0/3721274175 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f932804f470 con 0x7f93280d0ae0 2024-03-23T14:35:19.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.023+0000 7f931e7fc640 1 -- 172.21.15.161:0/3721274175 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f9318018090 con 0x7f93280d0ae0 2024-03-23T14:35:19.024 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-23T14:35:19.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.026+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f0d1815d040 con 0x7f0d1815f140 2024-03-23T14:35:19.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 -- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f930c06b810 msgr2=0x7f930c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f930c06b810 0x7f930c06dcd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9324034000 tx=0x7f9324008030 comp rx=0 tx=0).stop 2024-03-23T14:35:19.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 -- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93280d0ae0 msgr2=0x7f9328201470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93280d0ae0 0x7f9328201470 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f931800c780 tx=0x7f931800cc40 comp rx=0 tx=0).stop 2024-03-23T14:35:19.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 -- 172.21.15.161:0/3721274175 shutdown_connections 2024-03-23T14:35:19.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f930c06b810 0x7f930c06dcd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9328161c20 0x7f93281fa860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f932815e030 0x7f9328160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 --2- 172.21.15.161:0/3721274175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93280d0ae0 0x7f9328201470 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 -- 172.21.15.161:0/3721274175 >> 172.21.15.161:0/3721274175 conn(0x7f93280b5a10 msgr2=0x7f93280b37a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f0d0e7fc640 1 -- 172.21.15.161:0/895227175 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f0d1815d040 con 0x7f0d1815f140 2024-03-23T14:35:19.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 -- 172.21.15.161:0/3721274175 shutdown_connections 2024-03-23T14:35:19.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.027+0000 7f930bfff640 1 -- 172.21.15.161:0/3721274175 wait complete. 2024-03-23T14:35:19.026 INFO:teuthology.orchestra.run.smithi161.stdout:55834574854 2024-03-23T14:35:19.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.032+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0cfc06b8e0 msgr2=0x7f0cfc06dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.032+0000 7f0d1e899640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0cfc06b8e0 0x7f0cfc06dda0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0d14006fd0 tx=0x7f0d14008040 comp rx=0 tx=0).stop 2024-03-23T14:35:19.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.032+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1815f140 msgr2=0x7f0d18159780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.032+0000 7f0d1e899640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1815f140 0x7f0d18159780 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0d1000b1a0 tx=0x7f0d10004ab0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fa96c04f550 con 0x7fa96c15da90 2024-03-23T14:35:19.036 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.6 2024-03-23T14:35:19.037 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:19.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 shutdown_connections 2024-03-23T14:35:19.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7f0d1e899640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0cfc06b8e0 0x7f0cfc06dda0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7f0d1e899640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0d1815f140 0x7f0d18159780 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7f0d1e899640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0d18144160 0x7f0d18159240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7f0d1e899640 1 --2- 172.21.15.161:0/895227175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0d1806e520 0x7f0d1815d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 >> 172.21.15.161:0/895227175 conn(0x7f0d180b17f0 msgr2=0x7f0d180b0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 shutdown_connections 2024-03-23T14:35:19.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7f0d1e899640 1 -- 172.21.15.161:0/895227175 wait complete. 2024-03-23T14:35:19.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.033+0000 7fa9697fa640 1 -- 172.21.15.161:0/4260343870 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa964016090 con 0x7fa96c15da90 2024-03-23T14:35:19.038 INFO:teuthology.orchestra.run.smithi161.stdout:55834574856 2024-03-23T14:35:19.040 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.3 2024-03-23T14:35:19.040 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:19.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.037+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa95806b810 msgr2=0x7fa95806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.037+0000 7fa972bc2640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa95806b810 0x7fa95806dcd0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fa95c000c00 tx=0x7fa95c032000 comp rx=0 tx=0).stop 2024-03-23T14:35:19.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.037+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 msgr2=0x7fa96c1597e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.037+0000 7fa972bc2640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 0x7fa96c1597e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fa96400c780 tx=0x7fa96400cc40 comp rx=0 tx=0).stop 2024-03-23T14:35:19.042 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.7 2024-03-23T14:35:19.042 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:19.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.038+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 shutdown_connections 2024-03-23T14:35:19.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.038+0000 7fa972bc2640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa95806b810 0x7fa95806dcd0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.038+0000 7fa972bc2640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa96c15da90 0x7fa96c1597e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.038+0000 7fa972bc2640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa96c0cf5c0 0x7fa96c148a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.038+0000 7fa972bc2640 1 --2- 172.21.15.161:0/4260343870 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa96c0b3aa0 0x7fa96c1f8f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.038+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 >> 172.21.15.161:0/4260343870 conn(0x7fa96c0b17f0 msgr2=0x7fa96c0cd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.038+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 shutdown_connections 2024-03-23T14:35:19.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.038+0000 7fa972bc2640 1 -- 172.21.15.161:0/4260343870 wait complete. 2024-03-23T14:35:19.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-03-23T14:35:19.044 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:19.044 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-03-23T14:35:19.045 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:19.045 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.4 2024-03-23T14:35:19.045 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:19.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.5 2024-03-23T14:35:19.049 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:19.049 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.1 2024-03-23T14:35:19.049 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:19.050 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-23T14:35:19.050 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:35:19.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.195+0000 7f9245fa1640 1 Processor -- start 2024-03-23T14:35:19.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.195+0000 7f9245fa1640 1 -- start start 2024-03-23T14:35:19.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f9245fa1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9240145080 0x7f9240072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f9245fa1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92401607d0 0x7f924015fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f9245fa1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924006cbf0 0x7f9240160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f9245fa1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9240071010 con 0x7f92401607d0 2024-03-23T14:35:19.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f9245fa1640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f92400cf4f0 con 0x7f9240145080 2024-03-23T14:35:19.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f9245fa1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f92400cef30 con 0x7f924006cbf0 2024-03-23T14:35:19.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f923effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92401607d0 0x7f924015fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f923effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92401607d0 0x7f924015fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41324/0 (socket says 172.21.15.161:41324) 2024-03-23T14:35:19.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f923effd640 1 -- 172.21.15.161:0/421891245 learned_addr learned my addr 172.21.15.161:0/421891245 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:19.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f923ffff640 1 --2- 172.21.15.161:0/421891245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924006cbf0 0x7f9240160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f923f7fe640 1 --2- 172.21.15.161:0/421891245 >> v2:172.21.15.183:3300/0 conn(0x7f9240145080 0x7f9240072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f923effd640 1 -- 172.21.15.161:0/421891245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924006cbf0 msgr2=0x7f9240160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:19.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.196+0000 7f923effd640 1 --2- 172.21.15.161:0/421891245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924006cbf0 0x7f9240160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.197+0000 7f923effd640 1 -- 172.21.15.161:0/421891245 >> v2:172.21.15.183:3300/0 conn(0x7f9240145080 msgr2=0x7f9240072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.197+0000 7f923effd640 1 --2- 172.21.15.161:0/421891245 >> v2:172.21.15.183:3300/0 conn(0x7f9240145080 0x7f9240072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.197+0000 7f923effd640 1 -- 172.21.15.161:0/421891245 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92400cf210 con 0x7f92401607d0 2024-03-23T14:35:19.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.197+0000 7f923effd640 1 --2- 172.21.15.161:0/421891245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f924015fae0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f924006c9a0 tx=0x7f923402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a341d02ede90ad97 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.197+0000 7f923e7fc640 1 -- 172.21.15.161:0/421891245 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92340317e0 con 0x7f92401607d0 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.197+0000 7f923e7fc640 1 -- 172.21.15.161:0/421891245 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9234031940 con 0x7f92401607d0 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.197+0000 7f923e7fc640 1 -- 172.21.15.161:0/421891245 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9234031cd0 con 0x7f92401607d0 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 -- 172.21.15.161:0/421891245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 msgr2=0x7f924015fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 --2- 172.21.15.161:0/421891245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f924015fae0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f924006c9a0 tx=0x7f923402b880 comp rx=0 tx=0).stop 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 -- 172.21.15.161:0/421891245 shutdown_connections 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 --2- 172.21.15.161:0/421891245 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924006cbf0 0x7f9240160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 --2- 172.21.15.161:0/421891245 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f924015fae0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 --2- 172.21.15.161:0/421891245 >> v2:172.21.15.183:3300/0 conn(0x7f9240145080 0x7f9240072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 -- 172.21.15.161:0/421891245 >> 172.21.15.161:0/421891245 conn(0x7f92400b17f0 msgr2=0x7f92400b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 -- 172.21.15.161:0/421891245 shutdown_connections 2024-03-23T14:35:19.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.198+0000 7f9245fa1640 1 -- 172.21.15.161:0/421891245 wait complete. 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.199+0000 7f9245fa1640 1 Processor -- start 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.199+0000 7f9245fa1640 1 -- start start 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f9245fa1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f924006cbf0 0x7f9240143c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f9245fa1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9240145080 0x7f92401362f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f9245fa1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f92401f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f9245fa1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92401f5b50 con 0x7f92401607d0 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f9245fa1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f92401f4f40 con 0x7f924006cbf0 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f9245fa1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f92401f5220 con 0x7f9240145080 2024-03-23T14:35:19.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f92401f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f92401f0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41330/0 (socket says 172.21.15.161:41330) 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923ffff640 1 -- 172.21.15.161:0/768068983 learned_addr learned my addr 172.21.15.161:0/768068983 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923f7fe640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f924006cbf0 0x7f9240143c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923effd640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9240145080 0x7f92401362f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923ffff640 1 -- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9240145080 msgr2=0x7f92401362f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923ffff640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9240145080 0x7f92401362f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923ffff640 1 -- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f924006cbf0 msgr2=0x7f9240143c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923ffff640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f924006cbf0 0x7f9240143c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.200+0000 7f923ffff640 1 -- 172.21.15.161:0/768068983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92340094d0 con 0x7f92401607d0 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.201+0000 7f923ffff640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f92401f0ce0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f923800eed0 tx=0x7f923800b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.201+0000 7f9227fff640 1 -- 172.21.15.161:0/768068983 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238010040 con 0x7f92401607d0 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.201+0000 7f9227fff640 1 -- 172.21.15.161:0/768068983 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923800bd20 con 0x7f92401607d0 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.201+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92401f5500 con 0x7f92401607d0 2024-03-23T14:35:19.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.201+0000 7f9227fff640 1 -- 172.21.15.161:0/768068983 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9238014620 con 0x7f92401607d0 2024-03-23T14:35:19.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.201+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92401f27b0 con 0x7f92401607d0 2024-03-23T14:35:19.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.202+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92401f5cc0 con 0x7f92401607d0 2024-03-23T14:35:19.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.202+0000 7f9227fff640 1 -- 172.21.15.161:0/768068983 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9238014780 con 0x7f92401607d0 2024-03-23T14:35:19.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.202+0000 7f9227fff640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f922c06b810 0x7f922c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.203+0000 7f9227fff640 1 -- 172.21.15.161:0/768068983 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f923808aad0 con 0x7f92401607d0 2024-03-23T14:35:19.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.203+0000 7f923f7fe640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f922c06b810 0x7f922c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.203+0000 7f923f7fe640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f922c06b810 0x7f922c06dcd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9230009b40 tx=0x7f9230009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.207+0000 7f9227fff640 1 -- 172.21.15.161:0/768068983 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f92380555a0 con 0x7f92401607d0 2024-03-23T14:35:19.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.377+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9240145460 con 0x7f922c06b810 2024-03-23T14:35:19.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.384+0000 7f9227fff640 1 -- 172.21.15.161:0/768068983 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f9230011a60 con 0x7f922c06b810 2024-03-23T14:35:19.380 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:19.395 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:35:19.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f922c06b810 msgr2=0x7f922c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f922c06b810 0x7f922c06dcd0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f9230009b40 tx=0x7f9230009190 comp rx=0 tx=0).stop 2024-03-23T14:35:19.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 msgr2=0x7f92401f0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f92401f0ce0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f923800eed0 tx=0x7f923800b480 comp rx=0 tx=0).stop 2024-03-23T14:35:19.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 shutdown_connections 2024-03-23T14:35:19.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f922c06b810 0x7f922c06dcd0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92401607d0 0x7f92401f0ce0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9240145080 0x7f92401362f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 --2- 172.21.15.161:0/768068983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f924006cbf0 0x7f9240143c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.386+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 >> 172.21.15.161:0/768068983 conn(0x7f92400b17f0 msgr2=0x7f92400b0230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.387+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 shutdown_connections 2024-03-23T14:35:19.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.387+0000 7f9245fa1640 1 -- 172.21.15.161:0/768068983 wait complete. 2024-03-23T14:35:19.399 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-23T14:35:18.475194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219048,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755672,"statfs":{"total":773094113280,"available":772869808128,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001270"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991991+0000","last_change":"2024-03-23T14:35:13.778548+0000","last_active":"2024-03-23T14:35:14.991991+0000","last_peered":"2024-03-23T14:35:14.991991+0000","last_clean":"2024-03-23T14:35:14.991991+0000","last_became_active":"2024-03-23T14:35:13.778166+0000","last_became_peered":"2024-03-23T14:35:13.778166+0000","last_unstale":"2024-03-23T14:35:14.991991+0000","last_undegraded":"2024-03-23T14:35:14.991991+0000","last_fullsized":"2024-03-23T14:35:14.991991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:19.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774593+0000","last_change":"2024-03-23T14:35:13.788107+0000","last_active":"2024-03-23T14:35:14.774593+0000","last_peered":"2024-03-23T14:35:14.774593+0000","last_clean":"2024-03-23T14:35:14.774593+0000","last_became_active":"2024-03-23T14:35:13.787705+0000","last_became_peered":"2024-03-23T14:35:13.787705+0000","last_unstale":"2024-03-23T14:35:14.774593+0000","last_undegraded":"2024-03-23T14:35:14.774593+0000","last_fullsized":"2024-03-23T14:35:14.774593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991567+0000","last_change":"2024-03-23T14:35:13.783735+0000","last_active":"2024-03-23T14:35:14.991567+0000","last_peered":"2024-03-23T14:35:14.991567+0000","last_clean":"2024-03-23T14:35:14.991567+0000","last_became_active":"2024-03-23T14:35:13.783349+0000","last_became_peered":"2024-03-23T14:35:13.783349+0000","last_unstale":"2024-03-23T14:35:14.991567+0000","last_undegraded":"2024-03-23T14:35:14.991567+0000","last_fullsized":"2024-03-23T14:35:14.991567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991261+0000","last_change":"2024-03-23T14:35:13.787909+0000","last_active":"2024-03-23T14:35:14.991261+0000","last_peered":"2024-03-23T14:35:14.991261+0000","last_clean":"2024-03-23T14:35:14.991261+0000","last_became_active":"2024-03-23T14:35:13.786041+0000","last_became_peered":"2024-03-23T14:35:13.786041+0000","last_unstale":"2024-03-23T14:35:14.991261+0000","last_undegraded":"2024-03-23T14:35:14.991261+0000","last_fullsized":"2024-03-23T14:35:14.991261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991979+0000","last_change":"2024-03-23T14:35:13.784230+0000","last_active":"2024-03-23T14:35:14.991979+0000","last_peered":"2024-03-23T14:35:14.991979+0000","last_clean":"2024-03-23T14:35:14.991979+0000","last_became_active":"2024-03-23T14:35:13.781446+0000","last_became_peered":"2024-03-23T14:35:13.781446+0000","last_unstale":"2024-03-23T14:35:14.991979+0000","last_undegraded":"2024-03-23T14:35:14.991979+0000","last_fullsized":"2024-03-23T14:35:14.991979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:20.569093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991109+0000","last_change":"2024-03-23T14:35:13.785787+0000","last_active":"2024-03-23T14:35:14.991109+0000","last_peered":"2024-03-23T14:35:14.991109+0000","last_clean":"2024-03-23T14:35:14.991109+0000","last_became_active":"2024-03-23T14:35:13.785528+0000","last_became_peered":"2024-03-23T14:35:13.785528+0000","last_unstale":"2024-03-23T14:35:14.991109+0000","last_undegraded":"2024-03-23T14:35:14.991109+0000","last_fullsized":"2024-03-23T14:35:14.991109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991672+0000","last_change":"2024-03-23T14:35:13.787315+0000","last_active":"2024-03-23T14:35:14.991672+0000","last_peered":"2024-03-23T14:35:14.991672+0000","last_clean":"2024-03-23T14:35:14.991672+0000","last_became_active":"2024-03-23T14:35:13.787032+0000","last_became_peered":"2024-03-23T14:35:13.787032+0000","last_unstale":"2024-03-23T14:35:14.991672+0000","last_undegraded":"2024-03-23T14:35:14.991672+0000","last_fullsized":"2024-03-23T14:35:14.991672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991225+0000","last_change":"2024-03-23T14:35:13.779324+0000","last_active":"2024-03-23T14:35:14.991225+0000","last_peered":"2024-03-23T14:35:14.991225+0000","last_clean":"2024-03-23T14:35:14.991225+0000","last_became_active":"2024-03-23T14:35:13.779048+0000","last_became_peered":"2024-03-23T14:35:13.779048+0000","last_unstale":"2024-03-23T14:35:14.991225+0000","last_undegraded":"2024-03-23T14:35:14.991225+0000","last_fullsized":"2024-03-23T14:35:14.991225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:37.554855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992038+0000","last_change":"2024-03-23T14:35:13.781031+0000","last_active":"2024-03-23T14:35:14.992038+0000","last_peered":"2024-03-23T14:35:14.992038+0000","last_clean":"2024-03-23T14:35:14.992038+0000","last_became_active":"2024-03-23T14:35:13.780762+0000","last_became_peered":"2024-03-23T14:35:13.780762+0000","last_unstale":"2024-03-23T14:35:14.992038+0000","last_undegraded":"2024-03-23T14:35:14.992038+0000","last_fullsized":"2024-03-23T14:35:14.992038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:17:27.919253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916812+0000","last_change":"2024-03-23T14:35:13.776904+0000","last_active":"2024-03-23T14:35:14.916812+0000","last_peered":"2024-03-23T14:35:14.916812+0000","last_clean":"2024-03-23T14:35:14.916812+0000","last_became_active":"2024-03-23T14:35:13.776499+0000","last_became_peered":"2024-03-23T14:35:13.776499+0000","last_unstale":"2024-03-23T14:35:14.916812+0000","last_undegraded":"2024-03-23T14:35:14.916812+0000","last_fullsized":"2024-03-23T14:35:14.916812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916738+0000","last_change":"2024-03-23T14:35:13.777465+0000","last_active":"2024-03-23T14:35:14.916738+0000","last_peered":"2024-03-23T14:35:14.916738+0000","last_clean":"2024-03-23T14:35:14.916738+0000","last_became_active":"2024-03-23T14:35:13.777169+0000","last_became_peered":"2024-03-23T14:35:13.777169+0000","last_unstale":"2024-03-23T14:35:14.916738+0000","last_undegraded":"2024-03-23T14:35:14.916738+0000","last_fullsized":"2024-03-23T14:35:14.916738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775225+0000","last_change":"2024-03-23T14:35:13.776334+0000","last_active":"2024-03-23T14:35:14.775225+0000","last_peered":"2024-03-23T14:35:14.775225+0000","last_clean":"2024-03-23T14:35:14.775225+0000","last_became_active":"2024-03-23T14:35:13.776069+0000","last_became_peered":"2024-03-23T14:35:13.776069+0000","last_unstale":"2024-03-23T14:35:14.775225+0000","last_undegraded":"2024-03-23T14:35:14.775225+0000","last_fullsized":"2024-03-23T14:35:14.775225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:44.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991775+0000","last_change":"2024-03-23T14:35:13.779105+0000","last_active":"2024-03-23T14:35:14.991775+0000","last_peered":"2024-03-23T14:35:14.991775+0000","last_clean":"2024-03-23T14:35:14.991775+0000","last_became_active":"2024-03-23T14:35:13.778836+0000","last_became_peered":"2024-03-23T14:35:13.778836+0000","last_unstale":"2024-03-23T14:35:14.991775+0000","last_undegraded":"2024-03-23T14:35:14.991775+0000","last_fullsized":"2024-03-23T14:35:14.991775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991202+0000","last_change":"2024-03-23T14:35:13.786094+0000","last_active":"2024-03-23T14:35:14.991202+0000","last_peered":"2024-03-23T14:35:14.991202+0000","last_clean":"2024-03-23T14:35:14.991202+0000","last_became_active":"2024-03-23T14:35:13.785749+0000","last_became_peered":"2024-03-23T14:35:13.785749+0000","last_unstale":"2024-03-23T14:35:14.991202+0000","last_undegraded":"2024-03-23T14:35:14.991202+0000","last_fullsized":"2024-03-23T14:35:14.991202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:48:45.763326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774304+0000","last_change":"2024-03-23T14:35:13.763672+0000","last_active":"2024-03-23T14:35:14.774304+0000","last_peered":"2024-03-23T14:35:14.774304+0000","last_clean":"2024-03-23T14:35:14.774304+0000","last_became_active":"2024-03-23T14:35:13.763446+0000","last_became_peered":"2024-03-23T14:35:13.763446+0000","last_unstale":"2024-03-23T14:35:14.774304+0000","last_undegraded":"2024-03-23T14:35:14.774304+0000","last_fullsized":"2024-03-23T14:35:14.774304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.990985+0000","last_change":"2024-03-23T14:35:13.788261+0000","last_active":"2024-03-23T14:35:14.990985+0000","last_peered":"2024-03-23T14:35:14.990985+0000","last_clean":"2024-03-23T14:35:14.990985+0000","last_became_active":"2024-03-23T14:35:13.787931+0000","last_became_peered":"2024-03-23T14:35:13.787931+0000","last_unstale":"2024-03-23T14:35:14.990985+0000","last_undegraded":"2024-03-23T14:35:14.990985+0000","last_fullsized":"2024-03-23T14:35:14.990985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991568+0000","last_change":"2024-03-23T14:35:13.788504+0000","last_active":"2024-03-23T14:35:14.991568+0000","last_peered":"2024-03-23T14:35:14.991568+0000","last_clean":"2024-03-23T14:35:14.991568+0000","last_became_active":"2024-03-23T14:35:13.788220+0000","last_became_peered":"2024-03-23T14:35:13.788220+0000","last_unstale":"2024-03-23T14:35:14.991568+0000","last_undegraded":"2024-03-23T14:35:14.991568+0000","last_fullsized":"2024-03-23T14:35:14.991568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:25.033547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771552+0000","last_change":"2024-03-23T14:35:13.777872+0000","last_active":"2024-03-23T14:35:14.771552+0000","last_peered":"2024-03-23T14:35:14.771552+0000","last_clean":"2024-03-23T14:35:14.771552+0000","last_became_active":"2024-03-23T14:35:13.777618+0000","last_became_peered":"2024-03-23T14:35:13.777618+0000","last_unstale":"2024-03-23T14:35:14.771552+0000","last_undegraded":"2024-03-23T14:35:14.771552+0000","last_fullsized":"2024-03-23T14:35:14.771552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:51.829515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992132+0000","last_change":"2024-03-23T14:35:13.785636+0000","last_active":"2024-03-23T14:35:14.992132+0000","last_peered":"2024-03-23T14:35:14.992132+0000","last_clean":"2024-03-23T14:35:14.992132+0000","last_became_active":"2024-03-23T14:35:13.784525+0000","last_became_peered":"2024-03-23T14:35:13.784525+0000","last_unstale":"2024-03-23T14:35:14.992132+0000","last_undegraded":"2024-03-23T14:35:14.992132+0000","last_fullsized":"2024-03-23T14:35:14.992132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:25.033547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916605+0000","last_change":"2024-03-23T14:35:13.775546+0000","last_active":"2024-03-23T14:35:14.916605+0000","last_peered":"2024-03-23T14:35:14.916605+0000","last_clean":"2024-03-23T14:35:14.916605+0000","last_became_active":"2024-03-23T14:35:13.774705+0000","last_became_peered":"2024-03-23T14:35:13.774705+0000","last_unstale":"2024-03-23T14:35:14.916605+0000","last_undegraded":"2024-03-23T14:35:14.916605+0000","last_fullsized":"2024-03-23T14:35:14.916605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991370+0000","last_change":"2024-03-23T14:35:13.785890+0000","last_active":"2024-03-23T14:35:14.991370+0000","last_peered":"2024-03-23T14:35:14.991370+0000","last_clean":"2024-03-23T14:35:14.991370+0000","last_became_active":"2024-03-23T14:35:13.785564+0000","last_became_peered":"2024-03-23T14:35:13.785564+0000","last_unstale":"2024-03-23T14:35:14.991370+0000","last_undegraded":"2024-03-23T14:35:14.991370+0000","last_fullsized":"2024-03-23T14:35:14.991370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:16:34.110338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991141+0000","last_change":"2024-03-23T14:35:13.785106+0000","last_active":"2024-03-23T14:35:14.991141+0000","last_peered":"2024-03-23T14:35:14.991141+0000","last_clean":"2024-03-23T14:35:14.991141+0000","last_became_active":"2024-03-23T14:35:13.784543+0000","last_became_peered":"2024-03-23T14:35:13.784543+0000","last_unstale":"2024-03-23T14:35:14.991141+0000","last_undegraded":"2024-03-23T14:35:14.991141+0000","last_fullsized":"2024-03-23T14:35:14.991141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991371+0000","last_change":"2024-03-23T14:35:13.782489+0000","last_active":"2024-03-23T14:35:14.991371+0000","last_peered":"2024-03-23T14:35:14.991371+0000","last_clean":"2024-03-23T14:35:14.991371+0000","last_became_active":"2024-03-23T14:35:13.782292+0000","last_became_peered":"2024-03-23T14:35:13.782292+0000","last_unstale":"2024-03-23T14:35:14.991371+0000","last_undegraded":"2024-03-23T14:35:14.991371+0000","last_fullsized":"2024-03-23T14:35:14.991371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:16:34.110338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991174+0000","last_change":"2024-03-23T14:35:13.776346+0000","last_active":"2024-03-23T14:35:14.991174+0000","last_peered":"2024-03-23T14:35:14.991174+0000","last_clean":"2024-03-23T14:35:14.991174+0000","last_became_active":"2024-03-23T14:35:13.776085+0000","last_became_peered":"2024-03-23T14:35:13.776085+0000","last_unstale":"2024-03-23T14:35:14.991174+0000","last_undegraded":"2024-03-23T14:35:14.991174+0000","last_fullsized":"2024-03-23T14:35:14.991174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992164+0000","last_change":"2024-03-23T14:35:13.781666+0000","last_active":"2024-03-23T14:35:14.992164+0000","last_peered":"2024-03-23T14:35:14.992164+0000","last_clean":"2024-03-23T14:35:14.992164+0000","last_became_active":"2024-03-23T14:35:13.781380+0000","last_became_peered":"2024-03-23T14:35:13.781380+0000","last_unstale":"2024-03-23T14:35:14.992164+0000","last_undegraded":"2024-03-23T14:35:14.992164+0000","last_fullsized":"2024-03-23T14:35:14.992164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916837+0000","last_change":"2024-03-23T14:35:13.774246+0000","last_active":"2024-03-23T14:35:14.916837+0000","last_peered":"2024-03-23T14:35:14.916837+0000","last_clean":"2024-03-23T14:35:14.916837+0000","last_became_active":"2024-03-23T14:35:13.773945+0000","last_became_peered":"2024-03-23T14:35:13.773945+0000","last_unstale":"2024-03-23T14:35:14.916837+0000","last_undegraded":"2024-03-23T14:35:14.916837+0000","last_fullsized":"2024-03-23T14:35:14.916837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774384+0000","last_change":"2024-03-23T14:35:13.774154+0000","last_active":"2024-03-23T14:35:14.774384+0000","last_peered":"2024-03-23T14:35:14.774384+0000","last_clean":"2024-03-23T14:35:14.774384+0000","last_became_active":"2024-03-23T14:35:13.773320+0000","last_became_peered":"2024-03-23T14:35:13.773320+0000","last_unstale":"2024-03-23T14:35:14.774384+0000","last_undegraded":"2024-03-23T14:35:14.774384+0000","last_fullsized":"2024-03-23T14:35:14.774384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991890+0000","last_change":"2024-03-23T14:35:13.782131+0000","last_active":"2024-03-23T14:35:14.991890+0000","last_peered":"2024-03-23T14:35:14.991890+0000","last_clean":"2024-03-23T14:35:14.991890+0000","last_became_active":"2024-03-23T14:35:13.781455+0000","last_became_peered":"2024-03-23T14:35:13.781455+0000","last_unstale":"2024-03-23T14:35:14.991890+0000","last_undegraded":"2024-03-23T14:35:14.991890+0000","last_fullsized":"2024-03-23T14:35:14.991890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991510+0000","last_change":"2024-03-23T14:35:13.787101+0000","last_active":"2024-03-23T14:35:14.991510+0000","last_peered":"2024-03-23T14:35:14.991510+0000","last_clean":"2024-03-23T14:35:14.991510+0000","last_became_active":"2024-03-23T14:35:13.786742+0000","last_became_peered":"2024-03-23T14:35:13.786742+0000","last_unstale":"2024-03-23T14:35:14.991510+0000","last_undegraded":"2024-03-23T14:35:14.991510+0000","last_fullsized":"2024-03-23T14:35:14.991510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991510+0000","last_change":"2024-03-23T14:35:13.781923+0000","last_active":"2024-03-23T14:35:14.991510+0000","last_peered":"2024-03-23T14:35:14.991510+0000","last_clean":"2024-03-23T14:35:14.991510+0000","last_became_active":"2024-03-23T14:35:13.781553+0000","last_became_peered":"2024-03-23T14:35:13.781553+0000","last_unstale":"2024-03-23T14:35:14.991510+0000","last_undegraded":"2024-03-23T14:35:14.991510+0000","last_fullsized":"2024-03-23T14:35:14.991510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:31:20.667310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771511+0000","last_change":"2024-03-23T14:35:13.775171+0000","last_active":"2024-03-23T14:35:14.771511+0000","last_peered":"2024-03-23T14:35:14.771511+0000","last_clean":"2024-03-23T14:35:14.771511+0000","last_became_active":"2024-03-23T14:35:13.774898+0000","last_became_peered":"2024-03-23T14:35:13.774898+0000","last_unstale":"2024-03-23T14:35:14.771511+0000","last_undegraded":"2024-03-23T14:35:14.771511+0000","last_fullsized":"2024-03-23T14:35:14.771511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:44.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916695+0000","last_change":"2024-03-23T14:35:13.785730+0000","last_active":"2024-03-23T14:35:14.916695+0000","last_peered":"2024-03-23T14:35:14.916695+0000","last_clean":"2024-03-23T14:35:14.916695+0000","last_became_active":"2024-03-23T14:35:13.785427+0000","last_became_peered":"2024-03-23T14:35:13.785427+0000","last_unstale":"2024-03-23T14:35:14.916695+0000","last_undegraded":"2024-03-23T14:35:14.916695+0000","last_fullsized":"2024-03-23T14:35:14.916695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774946+0000","last_change":"2024-03-23T14:35:13.769227+0000","last_active":"2024-03-23T14:35:14.774946+0000","last_peered":"2024-03-23T14:35:14.774946+0000","last_clean":"2024-03-23T14:35:14.774946+0000","last_became_active":"2024-03-23T14:35:13.768866+0000","last_became_peered":"2024-03-23T14:35:13.768866+0000","last_unstale":"2024-03-23T14:35:14.774946+0000","last_undegraded":"2024-03-23T14:35:14.774946+0000","last_fullsized":"2024-03-23T14:35:14.774946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:26.689847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992031+0000","last_change":"2024-03-23T14:35:12.777511+0000","last_active":"2024-03-23T14:35:14.992031+0000","last_peered":"2024-03-23T14:35:14.992031+0000","last_clean":"2024-03-23T14:35:14.992031+0000","last_became_active":"2024-03-23T14:35:12.777173+0000","last_became_peered":"2024-03-23T14:35:12.777173+0000","last_unstale":"2024-03-23T14:35:14.992031+0000","last_undegraded":"2024-03-23T14:35:14.992031+0000","last_fullsized":"2024-03-23T14:35:14.992031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:50.295934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991649+0000","last_change":"2024-03-23T14:35:13.784656+0000","last_active":"2024-03-23T14:35:14.991649+0000","last_peered":"2024-03-23T14:35:14.991649+0000","last_clean":"2024-03-23T14:35:14.991649+0000","last_became_active":"2024-03-23T14:35:13.783372+0000","last_became_peered":"2024-03-23T14:35:13.783372+0000","last_unstale":"2024-03-23T14:35:14.991649+0000","last_undegraded":"2024-03-23T14:35:14.991649+0000","last_fullsized":"2024-03-23T14:35:14.991649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916556+0000","last_change":"2024-03-23T14:35:12.772860+0000","last_active":"2024-03-23T14:35:14.916556+0000","last_peered":"2024-03-23T14:35:14.916556+0000","last_clean":"2024-03-23T14:35:14.916556+0000","last_became_active":"2024-03-23T14:35:12.772437+0000","last_became_peered":"2024-03-23T14:35:12.772437+0000","last_unstale":"2024-03-23T14:35:14.916556+0000","last_undegraded":"2024-03-23T14:35:14.916556+0000","last_fullsized":"2024-03-23T14:35:14.916556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991525+0000","last_change":"2024-03-23T14:35:12.773627+0000","last_active":"2024-03-23T14:35:14.991525+0000","last_peered":"2024-03-23T14:35:14.991525+0000","last_clean":"2024-03-23T14:35:14.991525+0000","last_became_active":"2024-03-23T14:35:12.773323+0000","last_became_peered":"2024-03-23T14:35:12.773323+0000","last_unstale":"2024-03-23T14:35:14.991525+0000","last_undegraded":"2024-03-23T14:35:14.991525+0000","last_fullsized":"2024-03-23T14:35:14.991525+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991373+0000","last_change":"2024-03-23T14:35:13.786951+0000","last_active":"2024-03-23T14:35:14.991373+0000","last_peered":"2024-03-23T14:35:14.991373+0000","last_clean":"2024-03-23T14:35:14.991373+0000","last_became_active":"2024-03-23T14:35:13.786684+0000","last_became_peered":"2024-03-23T14:35:13.786684+0000","last_unstale":"2024-03-23T14:35:14.991373+0000","last_undegraded":"2024-03-23T14:35:14.991373+0000","last_fullsized":"2024-03-23T14:35:14.991373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991479+0000","last_change":"2024-03-23T14:35:12.775888+0000","last_active":"2024-03-23T14:35:14.991479+0000","last_peered":"2024-03-23T14:35:14.991479+0000","last_clean":"2024-03-23T14:35:14.991479+0000","last_became_active":"2024-03-23T14:35:12.775610+0000","last_became_peered":"2024-03-23T14:35:12.775610+0000","last_unstale":"2024-03-23T14:35:14.991479+0000","last_undegraded":"2024-03-23T14:35:14.991479+0000","last_fullsized":"2024-03-23T14:35:14.991479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774293+0000","last_change":"2024-03-23T14:35:13.778143+0000","last_active":"2024-03-23T14:35:14.774293+0000","last_peered":"2024-03-23T14:35:14.774293+0000","last_clean":"2024-03-23T14:35:14.774293+0000","last_became_active":"2024-03-23T14:35:13.776770+0000","last_became_peered":"2024-03-23T14:35:13.776770+0000","last_unstale":"2024-03-23T14:35:14.774293+0000","last_undegraded":"2024-03-23T14:35:14.774293+0000","last_fullsized":"2024-03-23T14:35:14.774293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916781+0000","last_change":"2024-03-23T14:35:12.776956+0000","last_active":"2024-03-23T14:35:14.916781+0000","last_peered":"2024-03-23T14:35:14.916781+0000","last_clean":"2024-03-23T14:35:14.916781+0000","last_became_active":"2024-03-23T14:35:12.776658+0000","last_became_peered":"2024-03-23T14:35:12.776658+0000","last_unstale":"2024-03-23T14:35:14.916781+0000","last_undegraded":"2024-03-23T14:35:14.916781+0000","last_fullsized":"2024-03-23T14:35:14.916781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991420+0000","last_change":"2024-03-23T14:35:13.786795+0000","last_active":"2024-03-23T14:35:14.991420+0000","last_peered":"2024-03-23T14:35:14.991420+0000","last_clean":"2024-03-23T14:35:14.991420+0000","last_became_active":"2024-03-23T14:35:13.786533+0000","last_became_peered":"2024-03-23T14:35:13.786533+0000","last_unstale":"2024-03-23T14:35:14.991420+0000","last_undegraded":"2024-03-23T14:35:14.991420+0000","last_fullsized":"2024-03-23T14:35:14.991420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:19.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991421+0000","last_change":"2024-03-23T14:35:12.779292+0000","last_active":"2024-03-23T14:35:14.991421+0000","last_peered":"2024-03-23T14:35:14.991421+0000","last_clean":"2024-03-23T14:35:14.991421+0000","last_became_active":"2024-03-23T14:35:12.778957+0000","last_became_peered":"2024-03-23T14:35:12.778957+0000","last_unstale":"2024-03-23T14:35:14.991421+0000","last_undegraded":"2024-03-23T14:35:14.991421+0000","last_fullsized":"2024-03-23T14:35:14.991421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991873+0000","last_change":"2024-03-23T14:35:13.785374+0000","last_active":"2024-03-23T14:35:14.991873+0000","last_peered":"2024-03-23T14:35:14.991873+0000","last_clean":"2024-03-23T14:35:14.991873+0000","last_became_active":"2024-03-23T14:35:13.784972+0000","last_became_peered":"2024-03-23T14:35:13.784972+0000","last_unstale":"2024-03-23T14:35:14.991873+0000","last_undegraded":"2024-03-23T14:35:14.991873+0000","last_fullsized":"2024-03-23T14:35:14.991873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774494+0000","last_change":"2024-03-23T14:35:12.775269+0000","last_active":"2024-03-23T14:35:14.774494+0000","last_peered":"2024-03-23T14:35:14.774494+0000","last_clean":"2024-03-23T14:35:14.774494+0000","last_became_active":"2024-03-23T14:35:12.774985+0000","last_became_peered":"2024-03-23T14:35:12.774985+0000","last_unstale":"2024-03-23T14:35:14.774494+0000","last_undegraded":"2024-03-23T14:35:14.774494+0000","last_fullsized":"2024-03-23T14:35:14.774494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775426+0000","last_change":"2024-03-23T14:35:13.777078+0000","last_active":"2024-03-23T14:35:14.775426+0000","last_peered":"2024-03-23T14:35:14.775426+0000","last_clean":"2024-03-23T14:35:14.775426+0000","last_became_active":"2024-03-23T14:35:13.776801+0000","last_became_peered":"2024-03-23T14:35:13.776801+0000","last_unstale":"2024-03-23T14:35:14.775426+0000","last_undegraded":"2024-03-23T14:35:14.775426+0000","last_fullsized":"2024-03-23T14:35:14.775426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:51.829515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916705+0000","last_change":"2024-03-23T14:35:12.773980+0000","last_active":"2024-03-23T14:35:14.916705+0000","last_peered":"2024-03-23T14:35:14.916705+0000","last_clean":"2024-03-23T14:35:14.916705+0000","last_became_active":"2024-03-23T14:35:12.773168+0000","last_became_peered":"2024-03-23T14:35:12.773168+0000","last_unstale":"2024-03-23T14:35:14.916705+0000","last_undegraded":"2024-03-23T14:35:14.916705+0000","last_fullsized":"2024-03-23T14:35:14.916705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991870+0000","last_change":"2024-03-23T14:35:13.789833+0000","last_active":"2024-03-23T14:35:14.991870+0000","last_peered":"2024-03-23T14:35:14.991870+0000","last_clean":"2024-03-23T14:35:14.991870+0000","last_became_active":"2024-03-23T14:35:13.789586+0000","last_became_peered":"2024-03-23T14:35:13.789586+0000","last_unstale":"2024-03-23T14:35:14.991870+0000","last_undegraded":"2024-03-23T14:35:14.991870+0000","last_fullsized":"2024-03-23T14:35:14.991870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991752+0000","last_change":"2024-03-23T14:35:12.773078+0000","last_active":"2024-03-23T14:35:14.991752+0000","last_peered":"2024-03-23T14:35:14.991752+0000","last_clean":"2024-03-23T14:35:14.991752+0000","last_became_active":"2024-03-23T14:35:12.772785+0000","last_became_peered":"2024-03-23T14:35:12.772785+0000","last_unstale":"2024-03-23T14:35:14.991752+0000","last_undegraded":"2024-03-23T14:35:14.991752+0000","last_fullsized":"2024-03-23T14:35:14.991752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:33:10.353085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991346+0000","last_change":"2024-03-23T14:35:13.784377+0000","last_active":"2024-03-23T14:35:14.991346+0000","last_peered":"2024-03-23T14:35:14.991346+0000","last_clean":"2024-03-23T14:35:14.991346+0000","last_became_active":"2024-03-23T14:35:13.783490+0000","last_became_peered":"2024-03-23T14:35:13.783490+0000","last_unstale":"2024-03-23T14:35:14.991346+0000","last_undegraded":"2024-03-23T14:35:14.991346+0000","last_fullsized":"2024-03-23T14:35:14.991346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:37.554855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991229+0000","last_change":"2024-03-23T14:35:12.780946+0000","last_active":"2024-03-23T14:35:14.991229+0000","last_peered":"2024-03-23T14:35:14.991229+0000","last_clean":"2024-03-23T14:35:14.991229+0000","last_became_active":"2024-03-23T14:35:12.780648+0000","last_became_peered":"2024-03-23T14:35:12.780648+0000","last_unstale":"2024-03-23T14:35:14.991229+0000","last_undegraded":"2024-03-23T14:35:14.991229+0000","last_fullsized":"2024-03-23T14:35:14.991229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992120+0000","last_change":"2024-03-23T14:35:13.784062+0000","last_active":"2024-03-23T14:35:14.992120+0000","last_peered":"2024-03-23T14:35:14.992120+0000","last_clean":"2024-03-23T14:35:14.992120+0000","last_became_active":"2024-03-23T14:35:13.783777+0000","last_became_peered":"2024-03-23T14:35:13.783777+0000","last_unstale":"2024-03-23T14:35:14.992120+0000","last_undegraded":"2024-03-23T14:35:14.992120+0000","last_fullsized":"2024-03-23T14:35:14.992120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:31:20.667310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916665+0000","last_change":"2024-03-23T14:35:12.772698+0000","last_active":"2024-03-23T14:35:14.916665+0000","last_peered":"2024-03-23T14:35:14.916665+0000","last_clean":"2024-03-23T14:35:14.916665+0000","last_became_active":"2024-03-23T14:35:12.772263+0000","last_became_peered":"2024-03-23T14:35:12.772263+0000","last_unstale":"2024-03-23T14:35:14.916665+0000","last_undegraded":"2024-03-23T14:35:14.916665+0000","last_fullsized":"2024-03-23T14:35:14.916665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916589+0000","last_change":"2024-03-23T14:35:13.785748+0000","last_active":"2024-03-23T14:35:14.916589+0000","last_peered":"2024-03-23T14:35:14.916589+0000","last_clean":"2024-03-23T14:35:14.916589+0000","last_became_active":"2024-03-23T14:35:13.785468+0000","last_became_peered":"2024-03-23T14:35:13.785468+0000","last_unstale":"2024-03-23T14:35:14.916589+0000","last_undegraded":"2024-03-23T14:35:14.916589+0000","last_fullsized":"2024-03-23T14:35:14.916589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771407+0000","last_change":"2024-03-23T14:35:12.766835+0000","last_active":"2024-03-23T14:35:14.771407+0000","last_peered":"2024-03-23T14:35:14.771407+0000","last_clean":"2024-03-23T14:35:14.771407+0000","last_became_active":"2024-03-23T14:35:12.766558+0000","last_became_peered":"2024-03-23T14:35:12.766558+0000","last_unstale":"2024-03-23T14:35:14.771407+0000","last_undegraded":"2024-03-23T14:35:14.771407+0000","last_fullsized":"2024-03-23T14:35:14.771407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:50.295934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991770+0000","last_change":"2024-03-23T14:35:13.783257+0000","last_active":"2024-03-23T14:35:14.991770+0000","last_peered":"2024-03-23T14:35:14.991770+0000","last_clean":"2024-03-23T14:35:14.991770+0000","last_became_active":"2024-03-23T14:35:13.782963+0000","last_became_peered":"2024-03-23T14:35:13.782963+0000","last_unstale":"2024-03-23T14:35:14.991770+0000","last_undegraded":"2024-03-23T14:35:14.991770+0000","last_fullsized":"2024-03-23T14:35:14.991770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775329+0000","last_change":"2024-03-23T14:35:12.769564+0000","last_active":"2024-03-23T14:35:14.775329+0000","last_peered":"2024-03-23T14:35:14.775329+0000","last_clean":"2024-03-23T14:35:14.775329+0000","last_became_active":"2024-03-23T14:35:12.769153+0000","last_became_peered":"2024-03-23T14:35:12.769153+0000","last_unstale":"2024-03-23T14:35:14.775329+0000","last_undegraded":"2024-03-23T14:35:14.775329+0000","last_fullsized":"2024-03-23T14:35:14.775329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:37:12.470551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991883+0000","last_change":"2024-03-23T14:35:13.783901+0000","last_active":"2024-03-23T14:35:14.991883+0000","last_peered":"2024-03-23T14:35:14.991883+0000","last_clean":"2024-03-23T14:35:14.991883+0000","last_became_active":"2024-03-23T14:35:13.783590+0000","last_became_peered":"2024-03-23T14:35:13.783590+0000","last_unstale":"2024-03-23T14:35:14.991883+0000","last_undegraded":"2024-03-23T14:35:14.991883+0000","last_fullsized":"2024-03-23T14:35:14.991883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771642+0000","last_change":"2024-03-23T14:35:12.767718+0000","last_active":"2024-03-23T14:35:14.771642+0000","last_peered":"2024-03-23T14:35:14.771642+0000","last_clean":"2024-03-23T14:35:14.771642+0000","last_became_active":"2024-03-23T14:35:12.767437+0000","last_became_peered":"2024-03-23T14:35:12.767437+0000","last_unstale":"2024-03-23T14:35:14.771642+0000","last_undegraded":"2024-03-23T14:35:14.771642+0000","last_fullsized":"2024-03-23T14:35:14.771642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992143+0000","last_change":"2024-03-23T14:35:13.785178+0000","last_active":"2024-03-23T14:35:14.992143+0000","last_peered":"2024-03-23T14:35:14.992143+0000","last_clean":"2024-03-23T14:35:14.992143+0000","last_became_active":"2024-03-23T14:35:13.781728+0000","last_became_peered":"2024-03-23T14:35:13.781728+0000","last_unstale":"2024-03-23T14:35:14.992143+0000","last_undegraded":"2024-03-23T14:35:14.992143+0000","last_fullsized":"2024-03-23T14:35:14.992143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:26.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774530+0000","last_change":"2024-03-23T14:35:12.764015+0000","last_active":"2024-03-23T14:35:14.774530+0000","last_peered":"2024-03-23T14:35:14.774530+0000","last_clean":"2024-03-23T14:35:14.774530+0000","last_became_active":"2024-03-23T14:35:12.763735+0000","last_became_peered":"2024-03-23T14:35:12.763735+0000","last_unstale":"2024-03-23T14:35:14.774530+0000","last_undegraded":"2024-03-23T14:35:14.774530+0000","last_fullsized":"2024-03-23T14:35:14.774530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:43.686652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991578+0000","last_change":"2024-03-23T14:35:13.780335+0000","last_active":"2024-03-23T14:35:14.991578+0000","last_peered":"2024-03-23T14:35:14.991578+0000","last_clean":"2024-03-23T14:35:14.991578+0000","last_became_active":"2024-03-23T14:35:13.780078+0000","last_became_peered":"2024-03-23T14:35:13.780078+0000","last_unstale":"2024-03-23T14:35:14.991578+0000","last_undegraded":"2024-03-23T14:35:14.991578+0000","last_fullsized":"2024-03-23T14:35:14.991578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:26.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771754+0000","last_change":"2024-03-23T14:35:12.772701+0000","last_active":"2024-03-23T14:35:14.771754+0000","last_peered":"2024-03-23T14:35:14.771754+0000","last_clean":"2024-03-23T14:35:14.771754+0000","last_became_active":"2024-03-23T14:35:12.772375+0000","last_became_peered":"2024-03-23T14:35:12.772375+0000","last_unstale":"2024-03-23T14:35:14.771754+0000","last_undegraded":"2024-03-23T14:35:14.771754+0000","last_fullsized":"2024-03-23T14:35:14.771754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991996+0000","last_change":"2024-03-23T14:35:13.789100+0000","last_active":"2024-03-23T14:35:14.991996+0000","last_peered":"2024-03-23T14:35:14.991996+0000","last_clean":"2024-03-23T14:35:14.991996+0000","last_became_active":"2024-03-23T14:35:13.788744+0000","last_became_peered":"2024-03-23T14:35:13.788744+0000","last_unstale":"2024-03-23T14:35:14.991996+0000","last_undegraded":"2024-03-23T14:35:14.991996+0000","last_fullsized":"2024-03-23T14:35:14.991996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:54:02.413325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991363+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:35:14.991363+0000","last_peered":"2024-03-23T14:35:14.991363+0000","last_clean":"2024-03-23T14:35:14.991363+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:35:14.991363+0000","last_undegraded":"2024-03-23T14:35:14.991363+0000","last_fullsized":"2024-03-23T14:35:14.991363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991027+0000","last_change":"2024-03-23T14:35:12.772428+0000","last_active":"2024-03-23T14:35:14.991027+0000","last_peered":"2024-03-23T14:35:14.991027+0000","last_clean":"2024-03-23T14:35:14.991027+0000","last_became_active":"2024-03-23T14:35:12.772152+0000","last_became_peered":"2024-03-23T14:35:12.772152+0000","last_unstale":"2024-03-23T14:35:14.991027+0000","last_undegraded":"2024-03-23T14:35:14.991027+0000","last_fullsized":"2024-03-23T14:35:14.991027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992036+0000","last_change":"2024-03-23T14:35:13.779880+0000","last_active":"2024-03-23T14:35:14.992036+0000","last_peered":"2024-03-23T14:35:14.992036+0000","last_clean":"2024-03-23T14:35:14.992036+0000","last_became_active":"2024-03-23T14:35:13.779608+0000","last_became_peered":"2024-03-23T14:35:13.779608+0000","last_unstale":"2024-03-23T14:35:14.992036+0000","last_undegraded":"2024-03-23T14:35:14.992036+0000","last_fullsized":"2024-03-23T14:35:14.992036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991307+0000","last_change":"2024-03-23T14:35:12.781319+0000","last_active":"2024-03-23T14:35:14.991307+0000","last_peered":"2024-03-23T14:35:14.991307+0000","last_clean":"2024-03-23T14:35:14.991307+0000","last_became_active":"2024-03-23T14:35:12.780393+0000","last_became_peered":"2024-03-23T14:35:12.780393+0000","last_unstale":"2024-03-23T14:35:14.991307+0000","last_undegraded":"2024-03-23T14:35:14.991307+0000","last_fullsized":"2024-03-23T14:35:14.991307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991191+0000","last_change":"2024-03-23T14:35:13.783755+0000","last_active":"2024-03-23T14:35:14.991191+0000","last_peered":"2024-03-23T14:35:14.991191+0000","last_clean":"2024-03-23T14:35:14.991191+0000","last_became_active":"2024-03-23T14:35:13.783415+0000","last_became_peered":"2024-03-23T14:35:13.783415+0000","last_unstale":"2024-03-23T14:35:14.991191+0000","last_undegraded":"2024-03-23T14:35:14.991191+0000","last_fullsized":"2024-03-23T14:35:14.991191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:07:11.094535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771472+0000","last_change":"2024-03-23T14:35:12.769935+0000","last_active":"2024-03-23T14:35:14.771472+0000","last_peered":"2024-03-23T14:35:14.771472+0000","last_clean":"2024-03-23T14:35:14.771472+0000","last_became_active":"2024-03-23T14:35:12.769638+0000","last_became_peered":"2024-03-23T14:35:12.769638+0000","last_unstale":"2024-03-23T14:35:14.771472+0000","last_undegraded":"2024-03-23T14:35:14.771472+0000","last_fullsized":"2024-03-23T14:35:14.771472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774313+0000","last_change":"2024-03-23T14:35:13.776295+0000","last_active":"2024-03-23T14:35:14.774313+0000","last_peered":"2024-03-23T14:35:14.774313+0000","last_clean":"2024-03-23T14:35:14.774313+0000","last_became_active":"2024-03-23T14:35:13.775968+0000","last_became_peered":"2024-03-23T14:35:13.775968+0000","last_unstale":"2024-03-23T14:35:14.774313+0000","last_undegraded":"2024-03-23T14:35:14.774313+0000","last_fullsized":"2024-03-23T14:35:14.774313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991315+0000","last_change":"2024-03-23T14:35:12.776276+0000","last_active":"2024-03-23T14:35:14.991315+0000","last_peered":"2024-03-23T14:35:14.991315+0000","last_clean":"2024-03-23T14:35:14.991315+0000","last_became_active":"2024-03-23T14:35:12.775990+0000","last_became_peered":"2024-03-23T14:35:12.775990+0000","last_unstale":"2024-03-23T14:35:14.991315+0000","last_undegraded":"2024-03-23T14:35:14.991315+0000","last_fullsized":"2024-03-23T14:35:14.991315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775094+0000","last_change":"2024-03-23T14:35:13.786554+0000","last_active":"2024-03-23T14:35:14.775094+0000","last_peered":"2024-03-23T14:35:14.775094+0000","last_clean":"2024-03-23T14:35:14.775094+0000","last_became_active":"2024-03-23T14:35:13.786286+0000","last_became_peered":"2024-03-23T14:35:13.786286+0000","last_unstale":"2024-03-23T14:35:14.775094+0000","last_undegraded":"2024-03-23T14:35:14.775094+0000","last_fullsized":"2024-03-23T14:35:14.775094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:40.665771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774420+0000","last_change":"2024-03-23T14:35:12.764502+0000","last_active":"2024-03-23T14:35:14.774420+0000","last_peered":"2024-03-23T14:35:14.774420+0000","last_clean":"2024-03-23T14:35:14.774420+0000","last_became_active":"2024-03-23T14:35:12.764241+0000","last_became_peered":"2024-03-23T14:35:12.764241+0000","last_unstale":"2024-03-23T14:35:14.774420+0000","last_undegraded":"2024-03-23T14:35:14.774420+0000","last_fullsized":"2024-03-23T14:35:14.774420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991506+0000","last_change":"2024-03-23T14:35:13.775872+0000","last_active":"2024-03-23T14:35:14.991506+0000","last_peered":"2024-03-23T14:35:14.991506+0000","last_clean":"2024-03-23T14:35:14.991506+0000","last_became_active":"2024-03-23T14:35:13.775527+0000","last_became_peered":"2024-03-23T14:35:13.775527+0000","last_unstale":"2024-03-23T14:35:14.991506+0000","last_undegraded":"2024-03-23T14:35:14.991506+0000","last_fullsized":"2024-03-23T14:35:14.991506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:17:27.919253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992153+0000","last_change":"2024-03-23T14:35:12.780722+0000","last_active":"2024-03-23T14:35:14.992153+0000","last_peered":"2024-03-23T14:35:14.992153+0000","last_clean":"2024-03-23T14:35:14.992153+0000","last_became_active":"2024-03-23T14:35:12.780349+0000","last_became_peered":"2024-03-23T14:35:12.780349+0000","last_unstale":"2024-03-23T14:35:14.992153+0000","last_undegraded":"2024-03-23T14:35:14.992153+0000","last_fullsized":"2024-03-23T14:35:14.992153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:50.822404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774384+0000","last_change":"2024-03-23T14:35:13.776300+0000","last_active":"2024-03-23T14:35:14.774384+0000","last_peered":"2024-03-23T14:35:14.774384+0000","last_clean":"2024-03-23T14:35:14.774384+0000","last_became_active":"2024-03-23T14:35:13.776007+0000","last_became_peered":"2024-03-23T14:35:13.776007+0000","last_unstale":"2024-03-23T14:35:14.774384+0000","last_undegraded":"2024-03-23T14:35:14.774384+0000","last_fullsized":"2024-03-23T14:35:14.774384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:15.491322+0000","last_change":"2024-03-23T14:35:12.772616+0000","last_active":"2024-03-23T14:35:15.491322+0000","last_peered":"2024-03-23T14:35:15.491322+0000","last_clean":"2024-03-23T14:35:15.491322+0000","last_became_active":"2024-03-23T14:35:12.772237+0000","last_became_peered":"2024-03-23T14:35:12.772237+0000","last_unstale":"2024-03-23T14:35:15.491322+0000","last_undegraded":"2024-03-23T14:35:15.491322+0000","last_fullsized":"2024-03-23T14:35:15.491322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:50.822404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992233+0000","last_change":"2024-03-23T14:35:13.782070+0000","last_active":"2024-03-23T14:35:14.992233+0000","last_peered":"2024-03-23T14:35:14.992233+0000","last_clean":"2024-03-23T14:35:14.992233+0000","last_became_active":"2024-03-23T14:35:13.781798+0000","last_became_peered":"2024-03-23T14:35:13.781798+0000","last_unstale":"2024-03-23T14:35:14.992233+0000","last_undegraded":"2024-03-23T14:35:14.992233+0000","last_fullsized":"2024-03-23T14:35:14.992233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916782+0000","last_change":"2024-03-23T14:35:12.777518+0000","last_active":"2024-03-23T14:35:14.916782+0000","last_peered":"2024-03-23T14:35:14.916782+0000","last_clean":"2024-03-23T14:35:14.916782+0000","last_became_active":"2024-03-23T14:35:12.777242+0000","last_became_peered":"2024-03-23T14:35:12.777242+0000","last_unstale":"2024-03-23T14:35:14.916782+0000","last_undegraded":"2024-03-23T14:35:14.916782+0000","last_fullsized":"2024-03-23T14:35:14.916782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991622+0000","last_change":"2024-03-23T14:35:13.786237+0000","last_active":"2024-03-23T14:35:14.991622+0000","last_peered":"2024-03-23T14:35:14.991622+0000","last_clean":"2024-03-23T14:35:14.991622+0000","last_became_active":"2024-03-23T14:35:13.785920+0000","last_became_peered":"2024-03-23T14:35:13.785920+0000","last_unstale":"2024-03-23T14:35:14.991622+0000","last_undegraded":"2024-03-23T14:35:14.991622+0000","last_fullsized":"2024-03-23T14:35:14.991622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991213+0000","last_change":"2024-03-23T14:35:12.781097+0000","last_active":"2024-03-23T14:35:14.991213+0000","last_peered":"2024-03-23T14:35:14.991213+0000","last_clean":"2024-03-23T14:35:14.991213+0000","last_became_active":"2024-03-23T14:35:12.780408+0000","last_became_peered":"2024-03-23T14:35:12.780408+0000","last_unstale":"2024-03-23T14:35:14.991213+0000","last_undegraded":"2024-03-23T14:35:14.991213+0000","last_fullsized":"2024-03-23T14:35:14.991213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774978+0000","last_change":"2024-03-23T14:35:13.777707+0000","last_active":"2024-03-23T14:35:14.774978+0000","last_peered":"2024-03-23T14:35:14.774978+0000","last_clean":"2024-03-23T14:35:14.774978+0000","last_became_active":"2024-03-23T14:35:13.777470+0000","last_became_peered":"2024-03-23T14:35:13.777470+0000","last_unstale":"2024-03-23T14:35:14.774978+0000","last_undegraded":"2024-03-23T14:35:14.774978+0000","last_fullsized":"2024-03-23T14:35:14.774978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:39.242232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774400+0000","last_change":"2024-03-23T14:35:12.771937+0000","last_active":"2024-03-23T14:35:14.774400+0000","last_peered":"2024-03-23T14:35:14.774400+0000","last_clean":"2024-03-23T14:35:14.774400+0000","last_became_active":"2024-03-23T14:35:12.771560+0000","last_became_peered":"2024-03-23T14:35:12.771560+0000","last_unstale":"2024-03-23T14:35:14.774400+0000","last_undegraded":"2024-03-23T14:35:14.774400+0000","last_fullsized":"2024-03-23T14:35:14.774400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991517+0000","last_change":"2024-03-23T14:35:13.779913+0000","last_active":"2024-03-23T14:35:14.991517+0000","last_peered":"2024-03-23T14:35:14.991517+0000","last_clean":"2024-03-23T14:35:14.991517+0000","last_became_active":"2024-03-23T14:35:13.779667+0000","last_became_peered":"2024-03-23T14:35:13.779667+0000","last_unstale":"2024-03-23T14:35:14.991517+0000","last_undegraded":"2024-03-23T14:35:14.991517+0000","last_fullsized":"2024-03-23T14:35:14.991517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774464+0000","last_change":"2024-03-23T14:35:12.770228+0000","last_active":"2024-03-23T14:35:14.774464+0000","last_peered":"2024-03-23T14:35:14.774464+0000","last_clean":"2024-03-23T14:35:14.774464+0000","last_became_active":"2024-03-23T14:35:12.769962+0000","last_became_peered":"2024-03-23T14:35:12.769962+0000","last_unstale":"2024-03-23T14:35:14.774464+0000","last_undegraded":"2024-03-23T14:35:14.774464+0000","last_fullsized":"2024-03-23T14:35:14.774464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991918+0000","last_change":"2024-03-23T14:35:13.786710+0000","last_active":"2024-03-23T14:35:14.991918+0000","last_peered":"2024-03-23T14:35:14.991918+0000","last_clean":"2024-03-23T14:35:14.991918+0000","last_became_active":"2024-03-23T14:35:13.786454+0000","last_became_peered":"2024-03-23T14:35:13.786454+0000","last_unstale":"2024-03-23T14:35:14.991918+0000","last_undegraded":"2024-03-23T14:35:14.991918+0000","last_fullsized":"2024-03-23T14:35:14.991918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992224+0000","last_change":"2024-03-23T14:35:12.778699+0000","last_active":"2024-03-23T14:35:14.992224+0000","last_peered":"2024-03-23T14:35:14.992224+0000","last_clean":"2024-03-23T14:35:14.992224+0000","last_became_active":"2024-03-23T14:35:12.778416+0000","last_became_peered":"2024-03-23T14:35:12.778416+0000","last_unstale":"2024-03-23T14:35:14.992224+0000","last_undegraded":"2024-03-23T14:35:14.992224+0000","last_fullsized":"2024-03-23T14:35:14.992224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916656+0000","last_change":"2024-03-23T14:35:13.773983+0000","last_active":"2024-03-23T14:35:14.916656+0000","last_peered":"2024-03-23T14:35:14.916656+0000","last_clean":"2024-03-23T14:35:14.916656+0000","last_became_active":"2024-03-23T14:35:13.773689+0000","last_became_peered":"2024-03-23T14:35:13.773689+0000","last_unstale":"2024-03-23T14:35:14.916656+0000","last_undegraded":"2024-03-23T14:35:14.916656+0000","last_fullsized":"2024-03-23T14:35:14.916656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991352+0000","last_change":"2024-03-23T14:35:13.783572+0000","last_active":"2024-03-23T14:35:14.991352+0000","last_peered":"2024-03-23T14:35:14.991352+0000","last_clean":"2024-03-23T14:35:14.991352+0000","last_became_active":"2024-03-23T14:35:13.782224+0000","last_became_peered":"2024-03-23T14:35:13.782224+0000","last_unstale":"2024-03-23T14:35:14.991352+0000","last_undegraded":"2024-03-23T14:35:14.991352+0000","last_fullsized":"2024-03-23T14:35:14.991352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:20.569093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991763+0000","last_change":"2024-03-23T14:35:12.775487+0000","last_active":"2024-03-23T14:35:14.991763+0000","last_peered":"2024-03-23T14:35:14.991763+0000","last_clean":"2024-03-23T14:35:14.991763+0000","last_became_active":"2024-03-23T14:35:12.775208+0000","last_became_peered":"2024-03-23T14:35:12.775208+0000","last_unstale":"2024-03-23T14:35:14.991763+0000","last_undegraded":"2024-03-23T14:35:14.991763+0000","last_fullsized":"2024-03-23T14:35:14.991763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991431+0000","last_change":"2024-03-23T14:35:13.783879+0000","last_active":"2024-03-23T14:35:14.991431+0000","last_peered":"2024-03-23T14:35:14.991431+0000","last_clean":"2024-03-23T14:35:14.991431+0000","last_became_active":"2024-03-23T14:35:13.783451+0000","last_became_peered":"2024-03-23T14:35:13.783451+0000","last_unstale":"2024-03-23T14:35:14.991431+0000","last_undegraded":"2024-03-23T14:35:14.991431+0000","last_fullsized":"2024-03-23T14:35:14.991431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:54:02.413325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992060+0000","last_change":"2024-03-23T14:35:12.777520+0000","last_active":"2024-03-23T14:35:14.992060+0000","last_peered":"2024-03-23T14:35:14.992060+0000","last_clean":"2024-03-23T14:35:14.992060+0000","last_became_active":"2024-03-23T14:35:12.777254+0000","last_became_peered":"2024-03-23T14:35:12.777254+0000","last_unstale":"2024-03-23T14:35:14.992060+0000","last_undegraded":"2024-03-23T14:35:14.992060+0000","last_fullsized":"2024-03-23T14:35:14.992060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:43.686652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991341+0000","last_change":"2024-03-23T14:35:13.785243+0000","last_active":"2024-03-23T14:35:14.991341+0000","last_peered":"2024-03-23T14:35:14.991341+0000","last_clean":"2024-03-23T14:35:14.991341+0000","last_became_active":"2024-03-23T14:35:13.784972+0000","last_became_peered":"2024-03-23T14:35:13.784972+0000","last_unstale":"2024-03-23T14:35:14.991341+0000","last_undegraded":"2024-03-23T14:35:14.991341+0000","last_fullsized":"2024-03-23T14:35:14.991341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:24:06.436157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992114+0000","last_change":"2024-03-23T14:35:12.770618+0000","last_active":"2024-03-23T14:35:14.992114+0000","last_peered":"2024-03-23T14:35:14.992114+0000","last_clean":"2024-03-23T14:35:14.992114+0000","last_became_active":"2024-03-23T14:35:12.770320+0000","last_became_peered":"2024-03-23T14:35:12.770320+0000","last_unstale":"2024-03-23T14:35:14.992114+0000","last_undegraded":"2024-03-23T14:35:14.992114+0000","last_fullsized":"2024-03-23T14:35:14.992114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:33:10.353085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991383+0000","last_change":"2024-03-23T14:35:13.775981+0000","last_active":"2024-03-23T14:35:14.991383+0000","last_peered":"2024-03-23T14:35:14.991383+0000","last_clean":"2024-03-23T14:35:14.991383+0000","last_became_active":"2024-03-23T14:35:13.775695+0000","last_became_peered":"2024-03-23T14:35:13.775695+0000","last_unstale":"2024-03-23T14:35:14.991383+0000","last_undegraded":"2024-03-23T14:35:14.991383+0000","last_fullsized":"2024-03-23T14:35:14.991383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991901+0000","last_change":"2024-03-23T14:35:12.773386+0000","last_active":"2024-03-23T14:35:14.991901+0000","last_peered":"2024-03-23T14:35:14.991901+0000","last_clean":"2024-03-23T14:35:14.991901+0000","last_became_active":"2024-03-23T14:35:12.773104+0000","last_became_peered":"2024-03-23T14:35:12.773104+0000","last_unstale":"2024-03-23T14:35:14.991901+0000","last_undegraded":"2024-03-23T14:35:14.991901+0000","last_fullsized":"2024-03-23T14:35:14.991901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:34.363745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:35:19.404 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:35:19.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.508+0000 7f0dd309a640 1 Processor -- start 2024-03-23T14:35:19.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.508+0000 7f0dd309a640 1 -- start start 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd309a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd309a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0dcc06eca0 0x7f0dcc0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd309a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dcc15be70 con 0x7f0dcc0724e0 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd309a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0dcc15c430 con 0x7f0dcc06eca0 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd309a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0dcc15c150 con 0x7f0dcc072100 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd1637640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0dcc06eca0 0x7f0dcc0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd1637640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0dcc06eca0 0x7f0dcc0b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41338/0 (socket says 172.21.15.161:41338) 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd1637640 1 -- 172.21.15.161:0/2340631208 learned_addr learned my addr 172.21.15.161:0/2340631208 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd0e36640 1 --2- 172.21.15.161:0/2340631208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dcbfff640 1 --1- 172.21.15.161:0/2340631208 >> v1:172.21.15.183:6789/0 conn(0x7f0dcc0724e0 0x7f0dcc06e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33932/0 (socket says 172.21.15.161:33932) 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd1637640 1 -- 172.21.15.161:0/2340631208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 msgr2=0x7f0dcc15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:19.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd1637640 1 --2- 172.21.15.161:0/2340631208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc15b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd1637640 1 -- 172.21.15.161:0/2340631208 >> v1:172.21.15.183:6789/0 conn(0x7f0dcc0724e0 legacy=0x7f0dcc06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dd1637640 1 -- 172.21.15.161:0/2340631208 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dcc0b2c70 con 0x7f0dcc06eca0 2024-03-23T14:35:19.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.509+0000 7f0dcb7fe640 1 -- 172.21.15.161:0/2340631208 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2185309299 0 0) 0x7f0dcc15be70 con 0x7f0dcc0724e0 2024-03-23T14:35:19.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.510+0000 7f0dd1637640 1 --2- 172.21.15.161:0/2340631208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc06eca0 0x7f0dcc0b3470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009e50 tx=0x7f0dc402bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=928d4be6aab145ff server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.510+0000 7f0dcb7fe640 1 -- 172.21.15.161:0/2340631208 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc402cb00 con 0x7f0dcc06eca0 2024-03-23T14:35:19.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.510+0000 7f0dcb7fe640 1 -- 172.21.15.161:0/2340631208 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dc402cc60 con 0x7f0dcc06eca0 2024-03-23T14:35:19.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.510+0000 7f0dcb7fe640 1 -- 172.21.15.161:0/2340631208 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dc40343f0 con 0x7f0dcc06eca0 2024-03-23T14:35:19.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.511+0000 7f0dd309a640 1 -- 172.21.15.161:0/2340631208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc06eca0 msgr2=0x7f0dcc0b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.511+0000 7f0dd309a640 1 --2- 172.21.15.161:0/2340631208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc06eca0 0x7f0dcc0b3470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0dc4009e50 tx=0x7f0dc402bf60 comp rx=0 tx=0).stop 2024-03-23T14:35:19.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.511+0000 7f0dd309a640 1 -- 172.21.15.161:0/2340631208 shutdown_connections 2024-03-23T14:35:19.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.511+0000 7f0dd309a640 1 --2- 172.21.15.161:0/2340631208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc06eca0 0x7f0dcc0b3470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.511+0000 7f0dd309a640 1 --2- 172.21.15.161:0/2340631208 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc15b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.511+0000 7f0dd309a640 1 -- 172.21.15.161:0/2340631208 >> 172.21.15.161:0/2340631208 conn(0x7f0dcc0b17f0 msgr2=0x7f0dcc0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.511+0000 7f0dd309a640 1 -- 172.21.15.161:0/2340631208 shutdown_connections 2024-03-23T14:35:19.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.511+0000 7f0dd309a640 1 -- 172.21.15.161:0/2340631208 wait complete. 2024-03-23T14:35:19.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.512+0000 7f0dd309a640 1 Processor -- start 2024-03-23T14:35:19.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.512+0000 7f0dd309a640 1 -- start start 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd309a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc06eca0 0x7f0dcc0cf150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd309a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd309a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc0724e0 0x7f0dcc14a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd309a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc0cf690 con 0x7f0dcc0724e0 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd309a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0dcc152260 con 0x7f0dcc06eca0 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd309a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0dcc0cde30 con 0x7f0dcc072100 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd1637640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc0724e0 0x7f0dcc14a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd0e36640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc06eca0 0x7f0dcc0cf150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dcbfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd1637640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc0724e0 0x7f0dcc14a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41340/0 (socket says 172.21.15.161:41340) 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd1637640 1 -- 172.21.15.161:0/1063777239 learned_addr learned my addr 172.21.15.161:0/1063777239 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dcbfff640 1 -- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc06eca0 msgr2=0x7f0dcc0cf150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dcbfff640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc06eca0 0x7f0dcc0cf150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dcbfff640 1 -- 172.21.15.161:0/1063777239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc0724e0 msgr2=0x7f0dcc14a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dcbfff640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc0724e0 0x7f0dcc14a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dcbfff640 1 -- 172.21.15.161:0/1063777239 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dc4009b30 con 0x7f0dcc072100 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd1637640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc0724e0 0x7f0dcc14a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd0e36640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc06eca0 0x7f0dcc0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dcbfff640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc151d20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0db800c410 tx=0x7f0db800c8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dc97fa640 1 -- 172.21.15.161:0/1063777239 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db80074e0 con 0x7f0dcc072100 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dc97fa640 1 -- 172.21.15.161:0/1063777239 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0db8007640 con 0x7f0dcc072100 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dcc14a6a0 con 0x7f0dcc072100 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.513+0000 7f0dc97fa640 1 -- 172.21.15.161:0/1063777239 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0db8004a20 con 0x7f0dcc072100 2024-03-23T14:35:19.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.514+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dcc14ce10 con 0x7f0dcc072100 2024-03-23T14:35:19.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.514+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dcc0735e0 con 0x7f0dcc072100 2024-03-23T14:35:19.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.515+0000 7f0dc97fa640 1 -- 172.21.15.161:0/1063777239 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0db8004b80 con 0x7f0dcc072100 2024-03-23T14:35:19.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.515+0000 7f0dc97fa640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0db006b810 0x7f0db006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.516+0000 7f0dc97fa640 1 -- 172.21.15.161:0/1063777239 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0db8089e30 con 0x7f0dcc072100 2024-03-23T14:35:19.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.516+0000 7f0dd0e36640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0db006b810 0x7f0db006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.516+0000 7f0dd0e36640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0db006b810 0x7f0db006dcd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc165d00 tx=0x7f0dc0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.524+0000 7f0dc97fa640 1 -- 172.21.15.161:0/1063777239 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0db80549d0 con 0x7f0dcc072100 2024-03-23T14:35:19.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.691+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0dcc0d0960 con 0x7f0db006b810 2024-03-23T14:35:19.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.697+0000 7f0dc97fa640 1 -- 172.21.15.161:0/1063777239 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251836 (secure 0 0 0) 0x7f0dcc0d0960 con 0x7f0db006b810 2024-03-23T14:35:19.694 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:19.706 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:35:19.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0db006b810 msgr2=0x7f0db006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0db006b810 0x7f0db006dcd0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0dcc165d00 tx=0x7f0dc0009190 comp rx=0 tx=0).stop 2024-03-23T14:35:19.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 msgr2=0x7f0dcc151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc151d20 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f0db800c410 tx=0x7f0db800c8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 shutdown_connections 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0db006b810 0x7f0db006dcd0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0dcc0724e0 0x7f0dcc14a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0dcc072100 0x7f0dcc151d20 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 --2- 172.21.15.161:0/1063777239 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0dcc06eca0 0x7f0dcc0cf150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 >> 172.21.15.161:0/1063777239 conn(0x7f0dcc0b17f0 msgr2=0x7f0dcc0d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 shutdown_connections 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.700+0000 7f0dd309a640 1 -- 172.21.15.161:0/1063777239 wait complete. 2024-03-23T14:35:19.710 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":22,"stamp":"2024-03-23T14:35:18.475194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219048,"kb_used_data":2824,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755672,"statfs":{"total":773094113280,"available":772869808128,"internally_reserved":0,"allocated":2891776,"data_stored":1753028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001270"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991991+0000","last_change":"2024-03-23T14:35:13.778548+0000","last_active":"2024-03-23T14:35:14.991991+0000","last_peered":"2024-03-23T14:35:14.991991+0000","last_clean":"2024-03-23T14:35:14.991991+0000","last_became_active":"2024-03-23T14:35:13.778166+0000","last_became_peered":"2024-03-23T14:35:13.778166+0000","last_unstale":"2024-03-23T14:35:14.991991+0000","last_undegraded":"2024-03-23T14:35:14.991991+0000","last_fullsized":"2024-03-23T14:35:14.991991+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:19.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774593+0000","last_change":"2024-03-23T14:35:13.788107+0000","last_active":"2024-03-23T14:35:14.774593+0000","last_peered":"2024-03-23T14:35:14.774593+0000","last_clean":"2024-03-23T14:35:14.774593+0000","last_became_active":"2024-03-23T14:35:13.787705+0000","last_became_peered":"2024-03-23T14:35:13.787705+0000","last_unstale":"2024-03-23T14:35:14.774593+0000","last_undegraded":"2024-03-23T14:35:14.774593+0000","last_fullsized":"2024-03-23T14:35:14.774593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991567+0000","last_change":"2024-03-23T14:35:13.783735+0000","last_active":"2024-03-23T14:35:14.991567+0000","last_peered":"2024-03-23T14:35:14.991567+0000","last_clean":"2024-03-23T14:35:14.991567+0000","last_became_active":"2024-03-23T14:35:13.783349+0000","last_became_peered":"2024-03-23T14:35:13.783349+0000","last_unstale":"2024-03-23T14:35:14.991567+0000","last_undegraded":"2024-03-23T14:35:14.991567+0000","last_fullsized":"2024-03-23T14:35:14.991567+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991261+0000","last_change":"2024-03-23T14:35:13.787909+0000","last_active":"2024-03-23T14:35:14.991261+0000","last_peered":"2024-03-23T14:35:14.991261+0000","last_clean":"2024-03-23T14:35:14.991261+0000","last_became_active":"2024-03-23T14:35:13.786041+0000","last_became_peered":"2024-03-23T14:35:13.786041+0000","last_unstale":"2024-03-23T14:35:14.991261+0000","last_undegraded":"2024-03-23T14:35:14.991261+0000","last_fullsized":"2024-03-23T14:35:14.991261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991979+0000","last_change":"2024-03-23T14:35:13.784230+0000","last_active":"2024-03-23T14:35:14.991979+0000","last_peered":"2024-03-23T14:35:14.991979+0000","last_clean":"2024-03-23T14:35:14.991979+0000","last_became_active":"2024-03-23T14:35:13.781446+0000","last_became_peered":"2024-03-23T14:35:13.781446+0000","last_unstale":"2024-03-23T14:35:14.991979+0000","last_undegraded":"2024-03-23T14:35:14.991979+0000","last_fullsized":"2024-03-23T14:35:14.991979+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:20.569093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991109+0000","last_change":"2024-03-23T14:35:13.785787+0000","last_active":"2024-03-23T14:35:14.991109+0000","last_peered":"2024-03-23T14:35:14.991109+0000","last_clean":"2024-03-23T14:35:14.991109+0000","last_became_active":"2024-03-23T14:35:13.785528+0000","last_became_peered":"2024-03-23T14:35:13.785528+0000","last_unstale":"2024-03-23T14:35:14.991109+0000","last_undegraded":"2024-03-23T14:35:14.991109+0000","last_fullsized":"2024-03-23T14:35:14.991109+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991672+0000","last_change":"2024-03-23T14:35:13.787315+0000","last_active":"2024-03-23T14:35:14.991672+0000","last_peered":"2024-03-23T14:35:14.991672+0000","last_clean":"2024-03-23T14:35:14.991672+0000","last_became_active":"2024-03-23T14:35:13.787032+0000","last_became_peered":"2024-03-23T14:35:13.787032+0000","last_unstale":"2024-03-23T14:35:14.991672+0000","last_undegraded":"2024-03-23T14:35:14.991672+0000","last_fullsized":"2024-03-23T14:35:14.991672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991225+0000","last_change":"2024-03-23T14:35:13.779324+0000","last_active":"2024-03-23T14:35:14.991225+0000","last_peered":"2024-03-23T14:35:14.991225+0000","last_clean":"2024-03-23T14:35:14.991225+0000","last_became_active":"2024-03-23T14:35:13.779048+0000","last_became_peered":"2024-03-23T14:35:13.779048+0000","last_unstale":"2024-03-23T14:35:14.991225+0000","last_undegraded":"2024-03-23T14:35:14.991225+0000","last_fullsized":"2024-03-23T14:35:14.991225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:37.554855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992038+0000","last_change":"2024-03-23T14:35:13.781031+0000","last_active":"2024-03-23T14:35:14.992038+0000","last_peered":"2024-03-23T14:35:14.992038+0000","last_clean":"2024-03-23T14:35:14.992038+0000","last_became_active":"2024-03-23T14:35:13.780762+0000","last_became_peered":"2024-03-23T14:35:13.780762+0000","last_unstale":"2024-03-23T14:35:14.992038+0000","last_undegraded":"2024-03-23T14:35:14.992038+0000","last_fullsized":"2024-03-23T14:35:14.992038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:17:27.919253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916812+0000","last_change":"2024-03-23T14:35:13.776904+0000","last_active":"2024-03-23T14:35:14.916812+0000","last_peered":"2024-03-23T14:35:14.916812+0000","last_clean":"2024-03-23T14:35:14.916812+0000","last_became_active":"2024-03-23T14:35:13.776499+0000","last_became_peered":"2024-03-23T14:35:13.776499+0000","last_unstale":"2024-03-23T14:35:14.916812+0000","last_undegraded":"2024-03-23T14:35:14.916812+0000","last_fullsized":"2024-03-23T14:35:14.916812+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916738+0000","last_change":"2024-03-23T14:35:13.777465+0000","last_active":"2024-03-23T14:35:14.916738+0000","last_peered":"2024-03-23T14:35:14.916738+0000","last_clean":"2024-03-23T14:35:14.916738+0000","last_became_active":"2024-03-23T14:35:13.777169+0000","last_became_peered":"2024-03-23T14:35:13.777169+0000","last_unstale":"2024-03-23T14:35:14.916738+0000","last_undegraded":"2024-03-23T14:35:14.916738+0000","last_fullsized":"2024-03-23T14:35:14.916738+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775225+0000","last_change":"2024-03-23T14:35:13.776334+0000","last_active":"2024-03-23T14:35:14.775225+0000","last_peered":"2024-03-23T14:35:14.775225+0000","last_clean":"2024-03-23T14:35:14.775225+0000","last_became_active":"2024-03-23T14:35:13.776069+0000","last_became_peered":"2024-03-23T14:35:13.776069+0000","last_unstale":"2024-03-23T14:35:14.775225+0000","last_undegraded":"2024-03-23T14:35:14.775225+0000","last_fullsized":"2024-03-23T14:35:14.775225+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:44.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991775+0000","last_change":"2024-03-23T14:35:13.779105+0000","last_active":"2024-03-23T14:35:14.991775+0000","last_peered":"2024-03-23T14:35:14.991775+0000","last_clean":"2024-03-23T14:35:14.991775+0000","last_became_active":"2024-03-23T14:35:13.778836+0000","last_became_peered":"2024-03-23T14:35:13.778836+0000","last_unstale":"2024-03-23T14:35:14.991775+0000","last_undegraded":"2024-03-23T14:35:14.991775+0000","last_fullsized":"2024-03-23T14:35:14.991775+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991202+0000","last_change":"2024-03-23T14:35:13.786094+0000","last_active":"2024-03-23T14:35:14.991202+0000","last_peered":"2024-03-23T14:35:14.991202+0000","last_clean":"2024-03-23T14:35:14.991202+0000","last_became_active":"2024-03-23T14:35:13.785749+0000","last_became_peered":"2024-03-23T14:35:13.785749+0000","last_unstale":"2024-03-23T14:35:14.991202+0000","last_undegraded":"2024-03-23T14:35:14.991202+0000","last_fullsized":"2024-03-23T14:35:14.991202+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:48:45.763326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774304+0000","last_change":"2024-03-23T14:35:13.763672+0000","last_active":"2024-03-23T14:35:14.774304+0000","last_peered":"2024-03-23T14:35:14.774304+0000","last_clean":"2024-03-23T14:35:14.774304+0000","last_became_active":"2024-03-23T14:35:13.763446+0000","last_became_peered":"2024-03-23T14:35:13.763446+0000","last_unstale":"2024-03-23T14:35:14.774304+0000","last_undegraded":"2024-03-23T14:35:14.774304+0000","last_fullsized":"2024-03-23T14:35:14.774304+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.990985+0000","last_change":"2024-03-23T14:35:13.788261+0000","last_active":"2024-03-23T14:35:14.990985+0000","last_peered":"2024-03-23T14:35:14.990985+0000","last_clean":"2024-03-23T14:35:14.990985+0000","last_became_active":"2024-03-23T14:35:13.787931+0000","last_became_peered":"2024-03-23T14:35:13.787931+0000","last_unstale":"2024-03-23T14:35:14.990985+0000","last_undegraded":"2024-03-23T14:35:14.990985+0000","last_fullsized":"2024-03-23T14:35:14.990985+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991568+0000","last_change":"2024-03-23T14:35:13.788504+0000","last_active":"2024-03-23T14:35:14.991568+0000","last_peered":"2024-03-23T14:35:14.991568+0000","last_clean":"2024-03-23T14:35:14.991568+0000","last_became_active":"2024-03-23T14:35:13.788220+0000","last_became_peered":"2024-03-23T14:35:13.788220+0000","last_unstale":"2024-03-23T14:35:14.991568+0000","last_undegraded":"2024-03-23T14:35:14.991568+0000","last_fullsized":"2024-03-23T14:35:14.991568+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:25.033547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771552+0000","last_change":"2024-03-23T14:35:13.777872+0000","last_active":"2024-03-23T14:35:14.771552+0000","last_peered":"2024-03-23T14:35:14.771552+0000","last_clean":"2024-03-23T14:35:14.771552+0000","last_became_active":"2024-03-23T14:35:13.777618+0000","last_became_peered":"2024-03-23T14:35:13.777618+0000","last_unstale":"2024-03-23T14:35:14.771552+0000","last_undegraded":"2024-03-23T14:35:14.771552+0000","last_fullsized":"2024-03-23T14:35:14.771552+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:51.829515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992132+0000","last_change":"2024-03-23T14:35:13.785636+0000","last_active":"2024-03-23T14:35:14.992132+0000","last_peered":"2024-03-23T14:35:14.992132+0000","last_clean":"2024-03-23T14:35:14.992132+0000","last_became_active":"2024-03-23T14:35:13.784525+0000","last_became_peered":"2024-03-23T14:35:13.784525+0000","last_unstale":"2024-03-23T14:35:14.992132+0000","last_undegraded":"2024-03-23T14:35:14.992132+0000","last_fullsized":"2024-03-23T14:35:14.992132+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:25.033547+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916605+0000","last_change":"2024-03-23T14:35:13.775546+0000","last_active":"2024-03-23T14:35:14.916605+0000","last_peered":"2024-03-23T14:35:14.916605+0000","last_clean":"2024-03-23T14:35:14.916605+0000","last_became_active":"2024-03-23T14:35:13.774705+0000","last_became_peered":"2024-03-23T14:35:13.774705+0000","last_unstale":"2024-03-23T14:35:14.916605+0000","last_undegraded":"2024-03-23T14:35:14.916605+0000","last_fullsized":"2024-03-23T14:35:14.916605+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991370+0000","last_change":"2024-03-23T14:35:13.785890+0000","last_active":"2024-03-23T14:35:14.991370+0000","last_peered":"2024-03-23T14:35:14.991370+0000","last_clean":"2024-03-23T14:35:14.991370+0000","last_became_active":"2024-03-23T14:35:13.785564+0000","last_became_peered":"2024-03-23T14:35:13.785564+0000","last_unstale":"2024-03-23T14:35:14.991370+0000","last_undegraded":"2024-03-23T14:35:14.991370+0000","last_fullsized":"2024-03-23T14:35:14.991370+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:16:34.110338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991141+0000","last_change":"2024-03-23T14:35:13.785106+0000","last_active":"2024-03-23T14:35:14.991141+0000","last_peered":"2024-03-23T14:35:14.991141+0000","last_clean":"2024-03-23T14:35:14.991141+0000","last_became_active":"2024-03-23T14:35:13.784543+0000","last_became_peered":"2024-03-23T14:35:13.784543+0000","last_unstale":"2024-03-23T14:35:14.991141+0000","last_undegraded":"2024-03-23T14:35:14.991141+0000","last_fullsized":"2024-03-23T14:35:14.991141+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991371+0000","last_change":"2024-03-23T14:35:13.782489+0000","last_active":"2024-03-23T14:35:14.991371+0000","last_peered":"2024-03-23T14:35:14.991371+0000","last_clean":"2024-03-23T14:35:14.991371+0000","last_became_active":"2024-03-23T14:35:13.782292+0000","last_became_peered":"2024-03-23T14:35:13.782292+0000","last_unstale":"2024-03-23T14:35:14.991371+0000","last_undegraded":"2024-03-23T14:35:14.991371+0000","last_fullsized":"2024-03-23T14:35:14.991371+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:16:34.110338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991174+0000","last_change":"2024-03-23T14:35:13.776346+0000","last_active":"2024-03-23T14:35:14.991174+0000","last_peered":"2024-03-23T14:35:14.991174+0000","last_clean":"2024-03-23T14:35:14.991174+0000","last_became_active":"2024-03-23T14:35:13.776085+0000","last_became_peered":"2024-03-23T14:35:13.776085+0000","last_unstale":"2024-03-23T14:35:14.991174+0000","last_undegraded":"2024-03-23T14:35:14.991174+0000","last_fullsized":"2024-03-23T14:35:14.991174+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992164+0000","last_change":"2024-03-23T14:35:13.781666+0000","last_active":"2024-03-23T14:35:14.992164+0000","last_peered":"2024-03-23T14:35:14.992164+0000","last_clean":"2024-03-23T14:35:14.992164+0000","last_became_active":"2024-03-23T14:35:13.781380+0000","last_became_peered":"2024-03-23T14:35:13.781380+0000","last_unstale":"2024-03-23T14:35:14.992164+0000","last_undegraded":"2024-03-23T14:35:14.992164+0000","last_fullsized":"2024-03-23T14:35:14.992164+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916837+0000","last_change":"2024-03-23T14:35:13.774246+0000","last_active":"2024-03-23T14:35:14.916837+0000","last_peered":"2024-03-23T14:35:14.916837+0000","last_clean":"2024-03-23T14:35:14.916837+0000","last_became_active":"2024-03-23T14:35:13.773945+0000","last_became_peered":"2024-03-23T14:35:13.773945+0000","last_unstale":"2024-03-23T14:35:14.916837+0000","last_undegraded":"2024-03-23T14:35:14.916837+0000","last_fullsized":"2024-03-23T14:35:14.916837+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774384+0000","last_change":"2024-03-23T14:35:13.774154+0000","last_active":"2024-03-23T14:35:14.774384+0000","last_peered":"2024-03-23T14:35:14.774384+0000","last_clean":"2024-03-23T14:35:14.774384+0000","last_became_active":"2024-03-23T14:35:13.773320+0000","last_became_peered":"2024-03-23T14:35:13.773320+0000","last_unstale":"2024-03-23T14:35:14.774384+0000","last_undegraded":"2024-03-23T14:35:14.774384+0000","last_fullsized":"2024-03-23T14:35:14.774384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991890+0000","last_change":"2024-03-23T14:35:13.782131+0000","last_active":"2024-03-23T14:35:14.991890+0000","last_peered":"2024-03-23T14:35:14.991890+0000","last_clean":"2024-03-23T14:35:14.991890+0000","last_became_active":"2024-03-23T14:35:13.781455+0000","last_became_peered":"2024-03-23T14:35:13.781455+0000","last_unstale":"2024-03-23T14:35:14.991890+0000","last_undegraded":"2024-03-23T14:35:14.991890+0000","last_fullsized":"2024-03-23T14:35:14.991890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991510+0000","last_change":"2024-03-23T14:35:13.787101+0000","last_active":"2024-03-23T14:35:14.991510+0000","last_peered":"2024-03-23T14:35:14.991510+0000","last_clean":"2024-03-23T14:35:14.991510+0000","last_became_active":"2024-03-23T14:35:13.786742+0000","last_became_peered":"2024-03-23T14:35:13.786742+0000","last_unstale":"2024-03-23T14:35:14.991510+0000","last_undegraded":"2024-03-23T14:35:14.991510+0000","last_fullsized":"2024-03-23T14:35:14.991510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991510+0000","last_change":"2024-03-23T14:35:13.781923+0000","last_active":"2024-03-23T14:35:14.991510+0000","last_peered":"2024-03-23T14:35:14.991510+0000","last_clean":"2024-03-23T14:35:14.991510+0000","last_became_active":"2024-03-23T14:35:13.781553+0000","last_became_peered":"2024-03-23T14:35:13.781553+0000","last_unstale":"2024-03-23T14:35:14.991510+0000","last_undegraded":"2024-03-23T14:35:14.991510+0000","last_fullsized":"2024-03-23T14:35:14.991510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:31:20.667310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771511+0000","last_change":"2024-03-23T14:35:13.775171+0000","last_active":"2024-03-23T14:35:14.771511+0000","last_peered":"2024-03-23T14:35:14.771511+0000","last_clean":"2024-03-23T14:35:14.771511+0000","last_became_active":"2024-03-23T14:35:13.774898+0000","last_became_peered":"2024-03-23T14:35:13.774898+0000","last_unstale":"2024-03-23T14:35:14.771511+0000","last_undegraded":"2024-03-23T14:35:14.771511+0000","last_fullsized":"2024-03-23T14:35:14.771511+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:44.693764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916695+0000","last_change":"2024-03-23T14:35:13.785730+0000","last_active":"2024-03-23T14:35:14.916695+0000","last_peered":"2024-03-23T14:35:14.916695+0000","last_clean":"2024-03-23T14:35:14.916695+0000","last_became_active":"2024-03-23T14:35:13.785427+0000","last_became_peered":"2024-03-23T14:35:13.785427+0000","last_unstale":"2024-03-23T14:35:14.916695+0000","last_undegraded":"2024-03-23T14:35:14.916695+0000","last_fullsized":"2024-03-23T14:35:14.916695+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774946+0000","last_change":"2024-03-23T14:35:13.769227+0000","last_active":"2024-03-23T14:35:14.774946+0000","last_peered":"2024-03-23T14:35:14.774946+0000","last_clean":"2024-03-23T14:35:14.774946+0000","last_became_active":"2024-03-23T14:35:13.768866+0000","last_became_peered":"2024-03-23T14:35:13.768866+0000","last_unstale":"2024-03-23T14:35:14.774946+0000","last_undegraded":"2024-03-23T14:35:14.774946+0000","last_fullsized":"2024-03-23T14:35:14.774946+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:26.689847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992031+0000","last_change":"2024-03-23T14:35:12.777511+0000","last_active":"2024-03-23T14:35:14.992031+0000","last_peered":"2024-03-23T14:35:14.992031+0000","last_clean":"2024-03-23T14:35:14.992031+0000","last_became_active":"2024-03-23T14:35:12.777173+0000","last_became_peered":"2024-03-23T14:35:12.777173+0000","last_unstale":"2024-03-23T14:35:14.992031+0000","last_undegraded":"2024-03-23T14:35:14.992031+0000","last_fullsized":"2024-03-23T14:35:14.992031+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:50.295934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991649+0000","last_change":"2024-03-23T14:35:13.784656+0000","last_active":"2024-03-23T14:35:14.991649+0000","last_peered":"2024-03-23T14:35:14.991649+0000","last_clean":"2024-03-23T14:35:14.991649+0000","last_became_active":"2024-03-23T14:35:13.783372+0000","last_became_peered":"2024-03-23T14:35:13.783372+0000","last_unstale":"2024-03-23T14:35:14.991649+0000","last_undegraded":"2024-03-23T14:35:14.991649+0000","last_fullsized":"2024-03-23T14:35:14.991649+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916556+0000","last_change":"2024-03-23T14:35:12.772860+0000","last_active":"2024-03-23T14:35:14.916556+0000","last_peered":"2024-03-23T14:35:14.916556+0000","last_clean":"2024-03-23T14:35:14.916556+0000","last_became_active":"2024-03-23T14:35:12.772437+0000","last_became_peered":"2024-03-23T14:35:12.772437+0000","last_unstale":"2024-03-23T14:35:14.916556+0000","last_undegraded":"2024-03-23T14:35:14.916556+0000","last_fullsized":"2024-03-23T14:35:14.916556+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991525+0000","last_change":"2024-03-23T14:35:12.773627+0000","last_active":"2024-03-23T14:35:14.991525+0000","last_peered":"2024-03-23T14:35:14.991525+0000","last_clean":"2024-03-23T14:35:14.991525+0000","last_became_active":"2024-03-23T14:35:12.773323+0000","last_became_peered":"2024-03-23T14:35:12.773323+0000","last_unstale":"2024-03-23T14:35:14.991525+0000","last_undegraded":"2024-03-23T14:35:14.991525+0000","last_fullsized":"2024-03-23T14:35:14.991525+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991373+0000","last_change":"2024-03-23T14:35:13.786951+0000","last_active":"2024-03-23T14:35:14.991373+0000","last_peered":"2024-03-23T14:35:14.991373+0000","last_clean":"2024-03-23T14:35:14.991373+0000","last_became_active":"2024-03-23T14:35:13.786684+0000","last_became_peered":"2024-03-23T14:35:13.786684+0000","last_unstale":"2024-03-23T14:35:14.991373+0000","last_undegraded":"2024-03-23T14:35:14.991373+0000","last_fullsized":"2024-03-23T14:35:14.991373+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991479+0000","last_change":"2024-03-23T14:35:12.775888+0000","last_active":"2024-03-23T14:35:14.991479+0000","last_peered":"2024-03-23T14:35:14.991479+0000","last_clean":"2024-03-23T14:35:14.991479+0000","last_became_active":"2024-03-23T14:35:12.775610+0000","last_became_peered":"2024-03-23T14:35:12.775610+0000","last_unstale":"2024-03-23T14:35:14.991479+0000","last_undegraded":"2024-03-23T14:35:14.991479+0000","last_fullsized":"2024-03-23T14:35:14.991479+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774293+0000","last_change":"2024-03-23T14:35:13.778143+0000","last_active":"2024-03-23T14:35:14.774293+0000","last_peered":"2024-03-23T14:35:14.774293+0000","last_clean":"2024-03-23T14:35:14.774293+0000","last_became_active":"2024-03-23T14:35:13.776770+0000","last_became_peered":"2024-03-23T14:35:13.776770+0000","last_unstale":"2024-03-23T14:35:14.774293+0000","last_undegraded":"2024-03-23T14:35:14.774293+0000","last_fullsized":"2024-03-23T14:35:14.774293+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916781+0000","last_change":"2024-03-23T14:35:12.776956+0000","last_active":"2024-03-23T14:35:14.916781+0000","last_peered":"2024-03-23T14:35:14.916781+0000","last_clean":"2024-03-23T14:35:14.916781+0000","last_became_active":"2024-03-23T14:35:12.776658+0000","last_became_peered":"2024-03-23T14:35:12.776658+0000","last_unstale":"2024-03-23T14:35:14.916781+0000","last_undegraded":"2024-03-23T14:35:14.916781+0000","last_fullsized":"2024-03-23T14:35:14.916781+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991420+0000","last_change":"2024-03-23T14:35:13.786795+0000","last_active":"2024-03-23T14:35:14.991420+0000","last_peered":"2024-03-23T14:35:14.991420+0000","last_clean":"2024-03-23T14:35:14.991420+0000","last_became_active":"2024-03-23T14:35:13.786533+0000","last_became_peered":"2024-03-23T14:35:13.786533+0000","last_unstale":"2024-03-23T14:35:14.991420+0000","last_undegraded":"2024-03-23T14:35:14.991420+0000","last_fullsized":"2024-03-23T14:35:14.991420+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:29:19.940282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991421+0000","last_change":"2024-03-23T14:35:12.779292+0000","last_active":"2024-03-23T14:35:14.991421+0000","last_peered":"2024-03-23T14:35:14.991421+0000","last_clean":"2024-03-23T14:35:14.991421+0000","last_became_active":"2024-03-23T14:35:12.778957+0000","last_became_peered":"2024-03-23T14:35:12.778957+0000","last_unstale":"2024-03-23T14:35:14.991421+0000","last_undegraded":"2024-03-23T14:35:14.991421+0000","last_fullsized":"2024-03-23T14:35:14.991421+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991873+0000","last_change":"2024-03-23T14:35:13.785374+0000","last_active":"2024-03-23T14:35:14.991873+0000","last_peered":"2024-03-23T14:35:14.991873+0000","last_clean":"2024-03-23T14:35:14.991873+0000","last_became_active":"2024-03-23T14:35:13.784972+0000","last_became_peered":"2024-03-23T14:35:13.784972+0000","last_unstale":"2024-03-23T14:35:14.991873+0000","last_undegraded":"2024-03-23T14:35:14.991873+0000","last_fullsized":"2024-03-23T14:35:14.991873+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"19'3","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774494+0000","last_change":"2024-03-23T14:35:12.775269+0000","last_active":"2024-03-23T14:35:14.774494+0000","last_peered":"2024-03-23T14:35:14.774494+0000","last_clean":"2024-03-23T14:35:14.774494+0000","last_became_active":"2024-03-23T14:35:12.774985+0000","last_became_peered":"2024-03-23T14:35:12.774985+0000","last_unstale":"2024-03-23T14:35:14.774494+0000","last_undegraded":"2024-03-23T14:35:14.774494+0000","last_fullsized":"2024-03-23T14:35:14.774494+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775426+0000","last_change":"2024-03-23T14:35:13.777078+0000","last_active":"2024-03-23T14:35:14.775426+0000","last_peered":"2024-03-23T14:35:14.775426+0000","last_clean":"2024-03-23T14:35:14.775426+0000","last_became_active":"2024-03-23T14:35:13.776801+0000","last_became_peered":"2024-03-23T14:35:13.776801+0000","last_unstale":"2024-03-23T14:35:14.775426+0000","last_undegraded":"2024-03-23T14:35:14.775426+0000","last_fullsized":"2024-03-23T14:35:14.775426+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:51.829515+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"19'6","reported_seq":20,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916705+0000","last_change":"2024-03-23T14:35:12.773980+0000","last_active":"2024-03-23T14:35:14.916705+0000","last_peered":"2024-03-23T14:35:14.916705+0000","last_clean":"2024-03-23T14:35:14.916705+0000","last_became_active":"2024-03-23T14:35:12.773168+0000","last_became_peered":"2024-03-23T14:35:12.773168+0000","last_unstale":"2024-03-23T14:35:14.916705+0000","last_undegraded":"2024-03-23T14:35:14.916705+0000","last_fullsized":"2024-03-23T14:35:14.916705+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991870+0000","last_change":"2024-03-23T14:35:13.789833+0000","last_active":"2024-03-23T14:35:14.991870+0000","last_peered":"2024-03-23T14:35:14.991870+0000","last_clean":"2024-03-23T14:35:14.991870+0000","last_became_active":"2024-03-23T14:35:13.789586+0000","last_became_peered":"2024-03-23T14:35:13.789586+0000","last_unstale":"2024-03-23T14:35:14.991870+0000","last_undegraded":"2024-03-23T14:35:14.991870+0000","last_fullsized":"2024-03-23T14:35:14.991870+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:41:59.587289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991752+0000","last_change":"2024-03-23T14:35:12.773078+0000","last_active":"2024-03-23T14:35:14.991752+0000","last_peered":"2024-03-23T14:35:14.991752+0000","last_clean":"2024-03-23T14:35:14.991752+0000","last_became_active":"2024-03-23T14:35:12.772785+0000","last_became_peered":"2024-03-23T14:35:12.772785+0000","last_unstale":"2024-03-23T14:35:14.991752+0000","last_undegraded":"2024-03-23T14:35:14.991752+0000","last_fullsized":"2024-03-23T14:35:14.991752+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:33:10.353085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991346+0000","last_change":"2024-03-23T14:35:13.784377+0000","last_active":"2024-03-23T14:35:14.991346+0000","last_peered":"2024-03-23T14:35:14.991346+0000","last_clean":"2024-03-23T14:35:14.991346+0000","last_became_active":"2024-03-23T14:35:13.783490+0000","last_became_peered":"2024-03-23T14:35:13.783490+0000","last_unstale":"2024-03-23T14:35:14.991346+0000","last_undegraded":"2024-03-23T14:35:14.991346+0000","last_fullsized":"2024-03-23T14:35:14.991346+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:14:37.554855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991229+0000","last_change":"2024-03-23T14:35:12.780946+0000","last_active":"2024-03-23T14:35:14.991229+0000","last_peered":"2024-03-23T14:35:14.991229+0000","last_clean":"2024-03-23T14:35:14.991229+0000","last_became_active":"2024-03-23T14:35:12.780648+0000","last_became_peered":"2024-03-23T14:35:12.780648+0000","last_unstale":"2024-03-23T14:35:14.991229+0000","last_undegraded":"2024-03-23T14:35:14.991229+0000","last_fullsized":"2024-03-23T14:35:14.991229+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992120+0000","last_change":"2024-03-23T14:35:13.784062+0000","last_active":"2024-03-23T14:35:14.992120+0000","last_peered":"2024-03-23T14:35:14.992120+0000","last_clean":"2024-03-23T14:35:14.992120+0000","last_became_active":"2024-03-23T14:35:13.783777+0000","last_became_peered":"2024-03-23T14:35:13.783777+0000","last_unstale":"2024-03-23T14:35:14.992120+0000","last_undegraded":"2024-03-23T14:35:14.992120+0000","last_fullsized":"2024-03-23T14:35:14.992120+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:31:20.667310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916665+0000","last_change":"2024-03-23T14:35:12.772698+0000","last_active":"2024-03-23T14:35:14.916665+0000","last_peered":"2024-03-23T14:35:14.916665+0000","last_clean":"2024-03-23T14:35:14.916665+0000","last_became_active":"2024-03-23T14:35:12.772263+0000","last_became_peered":"2024-03-23T14:35:12.772263+0000","last_unstale":"2024-03-23T14:35:14.916665+0000","last_undegraded":"2024-03-23T14:35:14.916665+0000","last_fullsized":"2024-03-23T14:35:14.916665+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916589+0000","last_change":"2024-03-23T14:35:13.785748+0000","last_active":"2024-03-23T14:35:14.916589+0000","last_peered":"2024-03-23T14:35:14.916589+0000","last_clean":"2024-03-23T14:35:14.916589+0000","last_became_active":"2024-03-23T14:35:13.785468+0000","last_became_peered":"2024-03-23T14:35:13.785468+0000","last_unstale":"2024-03-23T14:35:14.916589+0000","last_undegraded":"2024-03-23T14:35:14.916589+0000","last_fullsized":"2024-03-23T14:35:14.916589+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:00:53.459188+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"19'8","reported_seq":22,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771407+0000","last_change":"2024-03-23T14:35:12.766835+0000","last_active":"2024-03-23T14:35:14.771407+0000","last_peered":"2024-03-23T14:35:14.771407+0000","last_clean":"2024-03-23T14:35:14.771407+0000","last_became_active":"2024-03-23T14:35:12.766558+0000","last_became_peered":"2024-03-23T14:35:12.766558+0000","last_unstale":"2024-03-23T14:35:14.771407+0000","last_undegraded":"2024-03-23T14:35:14.771407+0000","last_fullsized":"2024-03-23T14:35:14.771407+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:02:50.295934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991770+0000","last_change":"2024-03-23T14:35:13.783257+0000","last_active":"2024-03-23T14:35:14.991770+0000","last_peered":"2024-03-23T14:35:14.991770+0000","last_clean":"2024-03-23T14:35:14.991770+0000","last_became_active":"2024-03-23T14:35:13.782963+0000","last_became_peered":"2024-03-23T14:35:13.782963+0000","last_unstale":"2024-03-23T14:35:14.991770+0000","last_undegraded":"2024-03-23T14:35:14.991770+0000","last_fullsized":"2024-03-23T14:35:14.991770+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775329+0000","last_change":"2024-03-23T14:35:12.769564+0000","last_active":"2024-03-23T14:35:14.775329+0000","last_peered":"2024-03-23T14:35:14.775329+0000","last_clean":"2024-03-23T14:35:14.775329+0000","last_became_active":"2024-03-23T14:35:12.769153+0000","last_became_peered":"2024-03-23T14:35:12.769153+0000","last_unstale":"2024-03-23T14:35:14.775329+0000","last_undegraded":"2024-03-23T14:35:14.775329+0000","last_fullsized":"2024-03-23T14:35:14.775329+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:37:12.470551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991883+0000","last_change":"2024-03-23T14:35:13.783901+0000","last_active":"2024-03-23T14:35:14.991883+0000","last_peered":"2024-03-23T14:35:14.991883+0000","last_clean":"2024-03-23T14:35:14.991883+0000","last_became_active":"2024-03-23T14:35:13.783590+0000","last_became_peered":"2024-03-23T14:35:13.783590+0000","last_unstale":"2024-03-23T14:35:14.991883+0000","last_undegraded":"2024-03-23T14:35:14.991883+0000","last_fullsized":"2024-03-23T14:35:14.991883+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771642+0000","last_change":"2024-03-23T14:35:12.767718+0000","last_active":"2024-03-23T14:35:14.771642+0000","last_peered":"2024-03-23T14:35:14.771642+0000","last_clean":"2024-03-23T14:35:14.771642+0000","last_became_active":"2024-03-23T14:35:12.767437+0000","last_became_peered":"2024-03-23T14:35:12.767437+0000","last_unstale":"2024-03-23T14:35:14.771642+0000","last_undegraded":"2024-03-23T14:35:14.771642+0000","last_fullsized":"2024-03-23T14:35:14.771642+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:33:24.970593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992143+0000","last_change":"2024-03-23T14:35:13.785178+0000","last_active":"2024-03-23T14:35:14.992143+0000","last_peered":"2024-03-23T14:35:14.992143+0000","last_clean":"2024-03-23T14:35:14.992143+0000","last_became_active":"2024-03-23T14:35:13.781728+0000","last_became_peered":"2024-03-23T14:35:13.781728+0000","last_unstale":"2024-03-23T14:35:14.992143+0000","last_undegraded":"2024-03-23T14:35:14.992143+0000","last_fullsized":"2024-03-23T14:35:14.992143+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:26.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774530+0000","last_change":"2024-03-23T14:35:12.764015+0000","last_active":"2024-03-23T14:35:14.774530+0000","last_peered":"2024-03-23T14:35:14.774530+0000","last_clean":"2024-03-23T14:35:14.774530+0000","last_became_active":"2024-03-23T14:35:12.763735+0000","last_became_peered":"2024-03-23T14:35:12.763735+0000","last_unstale":"2024-03-23T14:35:14.774530+0000","last_undegraded":"2024-03-23T14:35:14.774530+0000","last_fullsized":"2024-03-23T14:35:14.774530+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:43.686652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991578+0000","last_change":"2024-03-23T14:35:13.780335+0000","last_active":"2024-03-23T14:35:14.991578+0000","last_peered":"2024-03-23T14:35:14.991578+0000","last_clean":"2024-03-23T14:35:14.991578+0000","last_became_active":"2024-03-23T14:35:13.780078+0000","last_became_peered":"2024-03-23T14:35:13.780078+0000","last_unstale":"2024-03-23T14:35:14.991578+0000","last_undegraded":"2024-03-23T14:35:14.991578+0000","last_fullsized":"2024-03-23T14:35:14.991578+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:15:26.528794+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771754+0000","last_change":"2024-03-23T14:35:12.772701+0000","last_active":"2024-03-23T14:35:14.771754+0000","last_peered":"2024-03-23T14:35:14.771754+0000","last_clean":"2024-03-23T14:35:14.771754+0000","last_became_active":"2024-03-23T14:35:12.772375+0000","last_became_peered":"2024-03-23T14:35:12.772375+0000","last_unstale":"2024-03-23T14:35:14.771754+0000","last_undegraded":"2024-03-23T14:35:14.771754+0000","last_fullsized":"2024-03-23T14:35:14.771754+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991996+0000","last_change":"2024-03-23T14:35:13.789100+0000","last_active":"2024-03-23T14:35:14.991996+0000","last_peered":"2024-03-23T14:35:14.991996+0000","last_clean":"2024-03-23T14:35:14.991996+0000","last_became_active":"2024-03-23T14:35:13.788744+0000","last_became_peered":"2024-03-23T14:35:13.788744+0000","last_unstale":"2024-03-23T14:35:14.991996+0000","last_undegraded":"2024-03-23T14:35:14.991996+0000","last_fullsized":"2024-03-23T14:35:14.991996+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:54:02.413325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":170,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991363+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:35:14.991363+0000","last_peered":"2024-03-23T14:35:14.991363+0000","last_clean":"2024-03-23T14:35:14.991363+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:35:14.991363+0000","last_undegraded":"2024-03-23T14:35:14.991363+0000","last_fullsized":"2024-03-23T14:35:14.991363+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991027+0000","last_change":"2024-03-23T14:35:12.772428+0000","last_active":"2024-03-23T14:35:14.991027+0000","last_peered":"2024-03-23T14:35:14.991027+0000","last_clean":"2024-03-23T14:35:14.991027+0000","last_became_active":"2024-03-23T14:35:12.772152+0000","last_became_peered":"2024-03-23T14:35:12.772152+0000","last_unstale":"2024-03-23T14:35:14.991027+0000","last_undegraded":"2024-03-23T14:35:14.991027+0000","last_fullsized":"2024-03-23T14:35:14.991027+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992036+0000","last_change":"2024-03-23T14:35:13.779880+0000","last_active":"2024-03-23T14:35:14.992036+0000","last_peered":"2024-03-23T14:35:14.992036+0000","last_clean":"2024-03-23T14:35:14.992036+0000","last_became_active":"2024-03-23T14:35:13.779608+0000","last_became_peered":"2024-03-23T14:35:13.779608+0000","last_unstale":"2024-03-23T14:35:14.992036+0000","last_undegraded":"2024-03-23T14:35:14.992036+0000","last_fullsized":"2024-03-23T14:35:14.992036+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:47.828106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991307+0000","last_change":"2024-03-23T14:35:12.781319+0000","last_active":"2024-03-23T14:35:14.991307+0000","last_peered":"2024-03-23T14:35:14.991307+0000","last_clean":"2024-03-23T14:35:14.991307+0000","last_became_active":"2024-03-23T14:35:12.780393+0000","last_became_peered":"2024-03-23T14:35:12.780393+0000","last_unstale":"2024-03-23T14:35:14.991307+0000","last_undegraded":"2024-03-23T14:35:14.991307+0000","last_fullsized":"2024-03-23T14:35:14.991307+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991191+0000","last_change":"2024-03-23T14:35:13.783755+0000","last_active":"2024-03-23T14:35:14.991191+0000","last_peered":"2024-03-23T14:35:14.991191+0000","last_clean":"2024-03-23T14:35:14.991191+0000","last_became_active":"2024-03-23T14:35:13.783415+0000","last_became_peered":"2024-03-23T14:35:13.783415+0000","last_unstale":"2024-03-23T14:35:14.991191+0000","last_undegraded":"2024-03-23T14:35:14.991191+0000","last_fullsized":"2024-03-23T14:35:14.991191+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:07:11.094535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.771472+0000","last_change":"2024-03-23T14:35:12.769935+0000","last_active":"2024-03-23T14:35:14.771472+0000","last_peered":"2024-03-23T14:35:14.771472+0000","last_clean":"2024-03-23T14:35:14.771472+0000","last_became_active":"2024-03-23T14:35:12.769638+0000","last_became_peered":"2024-03-23T14:35:12.769638+0000","last_unstale":"2024-03-23T14:35:14.771472+0000","last_undegraded":"2024-03-23T14:35:14.771472+0000","last_fullsized":"2024-03-23T14:35:14.771472+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774313+0000","last_change":"2024-03-23T14:35:13.776295+0000","last_active":"2024-03-23T14:35:14.774313+0000","last_peered":"2024-03-23T14:35:14.774313+0000","last_clean":"2024-03-23T14:35:14.774313+0000","last_became_active":"2024-03-23T14:35:13.775968+0000","last_became_peered":"2024-03-23T14:35:13.775968+0000","last_unstale":"2024-03-23T14:35:14.774313+0000","last_undegraded":"2024-03-23T14:35:14.774313+0000","last_fullsized":"2024-03-23T14:35:14.774313+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991315+0000","last_change":"2024-03-23T14:35:12.776276+0000","last_active":"2024-03-23T14:35:14.991315+0000","last_peered":"2024-03-23T14:35:14.991315+0000","last_clean":"2024-03-23T14:35:14.991315+0000","last_became_active":"2024-03-23T14:35:12.775990+0000","last_became_peered":"2024-03-23T14:35:12.775990+0000","last_unstale":"2024-03-23T14:35:14.991315+0000","last_undegraded":"2024-03-23T14:35:14.991315+0000","last_fullsized":"2024-03-23T14:35:14.991315+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.775094+0000","last_change":"2024-03-23T14:35:13.786554+0000","last_active":"2024-03-23T14:35:14.775094+0000","last_peered":"2024-03-23T14:35:14.775094+0000","last_clean":"2024-03-23T14:35:14.775094+0000","last_became_active":"2024-03-23T14:35:13.786286+0000","last_became_peered":"2024-03-23T14:35:13.786286+0000","last_unstale":"2024-03-23T14:35:14.775094+0000","last_undegraded":"2024-03-23T14:35:14.775094+0000","last_fullsized":"2024-03-23T14:35:14.775094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:40.665771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774420+0000","last_change":"2024-03-23T14:35:12.764502+0000","last_active":"2024-03-23T14:35:14.774420+0000","last_peered":"2024-03-23T14:35:14.774420+0000","last_clean":"2024-03-23T14:35:14.774420+0000","last_became_active":"2024-03-23T14:35:12.764241+0000","last_became_peered":"2024-03-23T14:35:12.764241+0000","last_unstale":"2024-03-23T14:35:14.774420+0000","last_undegraded":"2024-03-23T14:35:14.774420+0000","last_fullsized":"2024-03-23T14:35:14.774420+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991506+0000","last_change":"2024-03-23T14:35:13.775872+0000","last_active":"2024-03-23T14:35:14.991506+0000","last_peered":"2024-03-23T14:35:14.991506+0000","last_clean":"2024-03-23T14:35:14.991506+0000","last_became_active":"2024-03-23T14:35:13.775527+0000","last_became_peered":"2024-03-23T14:35:13.775527+0000","last_unstale":"2024-03-23T14:35:14.991506+0000","last_undegraded":"2024-03-23T14:35:14.991506+0000","last_fullsized":"2024-03-23T14:35:14.991506+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:17:27.919253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992153+0000","last_change":"2024-03-23T14:35:12.780722+0000","last_active":"2024-03-23T14:35:14.992153+0000","last_peered":"2024-03-23T14:35:14.992153+0000","last_clean":"2024-03-23T14:35:14.992153+0000","last_became_active":"2024-03-23T14:35:12.780349+0000","last_became_peered":"2024-03-23T14:35:12.780349+0000","last_unstale":"2024-03-23T14:35:14.992153+0000","last_undegraded":"2024-03-23T14:35:14.992153+0000","last_fullsized":"2024-03-23T14:35:14.992153+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:50.822404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774384+0000","last_change":"2024-03-23T14:35:13.776300+0000","last_active":"2024-03-23T14:35:14.774384+0000","last_peered":"2024-03-23T14:35:14.774384+0000","last_clean":"2024-03-23T14:35:14.774384+0000","last_became_active":"2024-03-23T14:35:13.776007+0000","last_became_peered":"2024-03-23T14:35:13.776007+0000","last_unstale":"2024-03-23T14:35:14.774384+0000","last_undegraded":"2024-03-23T14:35:14.774384+0000","last_fullsized":"2024-03-23T14:35:14.774384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:21.960581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"20'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:15.491322+0000","last_change":"2024-03-23T14:35:12.772616+0000","last_active":"2024-03-23T14:35:15.491322+0000","last_peered":"2024-03-23T14:35:15.491322+0000","last_clean":"2024-03-23T14:35:15.491322+0000","last_became_active":"2024-03-23T14:35:12.772237+0000","last_became_peered":"2024-03-23T14:35:12.772237+0000","last_unstale":"2024-03-23T14:35:15.491322+0000","last_undegraded":"2024-03-23T14:35:15.491322+0000","last_fullsized":"2024-03-23T14:35:15.491322+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:50.822404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992233+0000","last_change":"2024-03-23T14:35:13.782070+0000","last_active":"2024-03-23T14:35:14.992233+0000","last_peered":"2024-03-23T14:35:14.992233+0000","last_clean":"2024-03-23T14:35:14.992233+0000","last_became_active":"2024-03-23T14:35:13.781798+0000","last_became_peered":"2024-03-23T14:35:13.781798+0000","last_unstale":"2024-03-23T14:35:14.992233+0000","last_undegraded":"2024-03-23T14:35:14.992233+0000","last_fullsized":"2024-03-23T14:35:14.992233+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:25:50.521248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916782+0000","last_change":"2024-03-23T14:35:12.777518+0000","last_active":"2024-03-23T14:35:14.916782+0000","last_peered":"2024-03-23T14:35:14.916782+0000","last_clean":"2024-03-23T14:35:14.916782+0000","last_became_active":"2024-03-23T14:35:12.777242+0000","last_became_peered":"2024-03-23T14:35:12.777242+0000","last_unstale":"2024-03-23T14:35:14.916782+0000","last_undegraded":"2024-03-23T14:35:14.916782+0000","last_fullsized":"2024-03-23T14:35:14.916782+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991622+0000","last_change":"2024-03-23T14:35:13.786237+0000","last_active":"2024-03-23T14:35:14.991622+0000","last_peered":"2024-03-23T14:35:14.991622+0000","last_clean":"2024-03-23T14:35:14.991622+0000","last_became_active":"2024-03-23T14:35:13.785920+0000","last_became_peered":"2024-03-23T14:35:13.785920+0000","last_unstale":"2024-03-23T14:35:14.991622+0000","last_undegraded":"2024-03-23T14:35:14.991622+0000","last_fullsized":"2024-03-23T14:35:14.991622+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991213+0000","last_change":"2024-03-23T14:35:12.781097+0000","last_active":"2024-03-23T14:35:14.991213+0000","last_peered":"2024-03-23T14:35:14.991213+0000","last_clean":"2024-03-23T14:35:14.991213+0000","last_became_active":"2024-03-23T14:35:12.780408+0000","last_became_peered":"2024-03-23T14:35:12.780408+0000","last_unstale":"2024-03-23T14:35:14.991213+0000","last_undegraded":"2024-03-23T14:35:14.991213+0000","last_fullsized":"2024-03-23T14:35:14.991213+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774978+0000","last_change":"2024-03-23T14:35:13.777707+0000","last_active":"2024-03-23T14:35:14.774978+0000","last_peered":"2024-03-23T14:35:14.774978+0000","last_clean":"2024-03-23T14:35:14.774978+0000","last_became_active":"2024-03-23T14:35:13.777470+0000","last_became_peered":"2024-03-23T14:35:13.777470+0000","last_unstale":"2024-03-23T14:35:14.774978+0000","last_undegraded":"2024-03-23T14:35:14.774978+0000","last_fullsized":"2024-03-23T14:35:14.774978+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:39.242232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774400+0000","last_change":"2024-03-23T14:35:12.771937+0000","last_active":"2024-03-23T14:35:14.774400+0000","last_peered":"2024-03-23T14:35:14.774400+0000","last_clean":"2024-03-23T14:35:14.774400+0000","last_became_active":"2024-03-23T14:35:12.771560+0000","last_became_peered":"2024-03-23T14:35:12.771560+0000","last_unstale":"2024-03-23T14:35:14.774400+0000","last_undegraded":"2024-03-23T14:35:14.774400+0000","last_fullsized":"2024-03-23T14:35:14.774400+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:28:11.750152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991517+0000","last_change":"2024-03-23T14:35:13.779913+0000","last_active":"2024-03-23T14:35:14.991517+0000","last_peered":"2024-03-23T14:35:14.991517+0000","last_clean":"2024-03-23T14:35:14.991517+0000","last_became_active":"2024-03-23T14:35:13.779667+0000","last_became_peered":"2024-03-23T14:35:13.779667+0000","last_unstale":"2024-03-23T14:35:14.991517+0000","last_undegraded":"2024-03-23T14:35:14.991517+0000","last_fullsized":"2024-03-23T14:35:14.991517+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:49:14.625766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.774464+0000","last_change":"2024-03-23T14:35:12.770228+0000","last_active":"2024-03-23T14:35:14.774464+0000","last_peered":"2024-03-23T14:35:14.774464+0000","last_clean":"2024-03-23T14:35:14.774464+0000","last_became_active":"2024-03-23T14:35:12.769962+0000","last_became_peered":"2024-03-23T14:35:12.769962+0000","last_unstale":"2024-03-23T14:35:14.774464+0000","last_undegraded":"2024-03-23T14:35:14.774464+0000","last_fullsized":"2024-03-23T14:35:14.774464+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:07:38.235121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991918+0000","last_change":"2024-03-23T14:35:13.786710+0000","last_active":"2024-03-23T14:35:14.991918+0000","last_peered":"2024-03-23T14:35:14.991918+0000","last_clean":"2024-03-23T14:35:14.991918+0000","last_became_active":"2024-03-23T14:35:13.786454+0000","last_became_peered":"2024-03-23T14:35:13.786454+0000","last_unstale":"2024-03-23T14:35:14.991918+0000","last_undegraded":"2024-03-23T14:35:14.991918+0000","last_fullsized":"2024-03-23T14:35:14.991918+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:19:52.569092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992224+0000","last_change":"2024-03-23T14:35:12.778699+0000","last_active":"2024-03-23T14:35:14.992224+0000","last_peered":"2024-03-23T14:35:14.992224+0000","last_clean":"2024-03-23T14:35:14.992224+0000","last_became_active":"2024-03-23T14:35:12.778416+0000","last_became_peered":"2024-03-23T14:35:12.778416+0000","last_unstale":"2024-03-23T14:35:14.992224+0000","last_undegraded":"2024-03-23T14:35:14.992224+0000","last_fullsized":"2024-03-23T14:35:14.992224+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:59:39.658659+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.916656+0000","last_change":"2024-03-23T14:35:13.773983+0000","last_active":"2024-03-23T14:35:14.916656+0000","last_peered":"2024-03-23T14:35:14.916656+0000","last_clean":"2024-03-23T14:35:14.916656+0000","last_became_active":"2024-03-23T14:35:13.773689+0000","last_became_peered":"2024-03-23T14:35:13.773689+0000","last_unstale":"2024-03-23T14:35:14.916656+0000","last_undegraded":"2024-03-23T14:35:14.916656+0000","last_fullsized":"2024-03-23T14:35:14.916656+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:32:37.533383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991352+0000","last_change":"2024-03-23T14:35:13.783572+0000","last_active":"2024-03-23T14:35:14.991352+0000","last_peered":"2024-03-23T14:35:14.991352+0000","last_clean":"2024-03-23T14:35:14.991352+0000","last_became_active":"2024-03-23T14:35:13.782224+0000","last_became_peered":"2024-03-23T14:35:13.782224+0000","last_unstale":"2024-03-23T14:35:14.991352+0000","last_undegraded":"2024-03-23T14:35:14.991352+0000","last_fullsized":"2024-03-23T14:35:14.991352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:34:20.569093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991763+0000","last_change":"2024-03-23T14:35:12.775487+0000","last_active":"2024-03-23T14:35:14.991763+0000","last_peered":"2024-03-23T14:35:14.991763+0000","last_clean":"2024-03-23T14:35:14.991763+0000","last_became_active":"2024-03-23T14:35:12.775208+0000","last_became_peered":"2024-03-23T14:35:12.775208+0000","last_unstale":"2024-03-23T14:35:14.991763+0000","last_undegraded":"2024-03-23T14:35:14.991763+0000","last_fullsized":"2024-03-23T14:35:14.991763+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:59:25.682736+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991431+0000","last_change":"2024-03-23T14:35:13.783879+0000","last_active":"2024-03-23T14:35:14.991431+0000","last_peered":"2024-03-23T14:35:14.991431+0000","last_clean":"2024-03-23T14:35:14.991431+0000","last_became_active":"2024-03-23T14:35:13.783451+0000","last_became_peered":"2024-03-23T14:35:13.783451+0000","last_unstale":"2024-03-23T14:35:14.991431+0000","last_undegraded":"2024-03-23T14:35:14.991431+0000","last_fullsized":"2024-03-23T14:35:14.991431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:54:02.413325+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"19'2","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992060+0000","last_change":"2024-03-23T14:35:12.777520+0000","last_active":"2024-03-23T14:35:14.992060+0000","last_peered":"2024-03-23T14:35:14.992060+0000","last_clean":"2024-03-23T14:35:14.992060+0000","last_became_active":"2024-03-23T14:35:12.777254+0000","last_became_peered":"2024-03-23T14:35:12.777254+0000","last_unstale":"2024-03-23T14:35:14.992060+0000","last_undegraded":"2024-03-23T14:35:14.992060+0000","last_fullsized":"2024-03-23T14:35:14.992060+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:13:43.686652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991341+0000","last_change":"2024-03-23T14:35:13.785243+0000","last_active":"2024-03-23T14:35:14.991341+0000","last_peered":"2024-03-23T14:35:14.991341+0000","last_clean":"2024-03-23T14:35:14.991341+0000","last_became_active":"2024-03-23T14:35:13.784972+0000","last_became_peered":"2024-03-23T14:35:13.784972+0000","last_unstale":"2024-03-23T14:35:14.991341+0000","last_undegraded":"2024-03-23T14:35:14.991341+0000","last_fullsized":"2024-03-23T14:35:14.991341+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:24:06.436157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"19'1","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.992114+0000","last_change":"2024-03-23T14:35:12.770618+0000","last_active":"2024-03-23T14:35:14.992114+0000","last_peered":"2024-03-23T14:35:14.992114+0000","last_clean":"2024-03-23T14:35:14.992114+0000","last_became_active":"2024-03-23T14:35:12.770320+0000","last_became_peered":"2024-03-23T14:35:12.770320+0000","last_unstale":"2024-03-23T14:35:14.992114+0000","last_undegraded":"2024-03-23T14:35:14.992114+0000","last_fullsized":"2024-03-23T14:35:14.992114+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:33:10.353085+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991383+0000","last_change":"2024-03-23T14:35:13.775981+0000","last_active":"2024-03-23T14:35:14.991383+0000","last_peered":"2024-03-23T14:35:14.991383+0000","last_clean":"2024-03-23T14:35:14.991383+0000","last_became_active":"2024-03-23T14:35:13.775695+0000","last_became_peered":"2024-03-23T14:35:13.775695+0000","last_unstale":"2024-03-23T14:35:14.991383+0000","last_undegraded":"2024-03-23T14:35:14.991383+0000","last_fullsized":"2024-03-23T14:35:14.991383+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:12.750635+0000","last_clean_scrub_stamp":"2024-03-23T14:35:12.750635+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:47:37.014324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"19'4","reported_seq":18,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-03-23T14:35:14.991901+0000","last_change":"2024-03-23T14:35:12.773386+0000","last_active":"2024-03-23T14:35:14.991901+0000","last_peered":"2024-03-23T14:35:14.991901+0000","last_clean":"2024-03-23T14:35:14.991901+0000","last_became_active":"2024-03-23T14:35:12.773104+0000","last_became_peered":"2024-03-23T14:35:12.773104+0000","last_unstale":"2024-03-23T14:35:14.991901+0000","last_undegraded":"2024-03-23T14:35:14.991901+0000","last_fullsized":"2024-03-23T14:35:14.991901+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:11.743523+0000","last_clean_scrub_stamp":"2024-03-23T14:35:11.743523+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:20:34.363745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574855,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27812,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344028,"statfs":{"total":96636764160,"available":96608284672,"internally_reserved":0,"allocated":802816,"data_stored":661884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":208896,"data_stored":71024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574856,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":71060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27240,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344600,"statfs":{"total":96636764160,"available":96608870400,"internally_reserved":0,"allocated":217088,"data_stored":72546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27228,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344612,"statfs":{"total":96636764160,"available":96608882688,"internally_reserved":0,"allocated":204800,"data_stored":70934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574856,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27236,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344604,"statfs":{"total":96636764160,"available":96608874496,"internally_reserved":0,"allocated":212992,"data_stored":71094,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27820,"kb_used_data":792,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344020,"statfs":{"total":96636764160,"available":96608276480,"internally_reserved":0,"allocated":811008,"data_stored":661940,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:35:19.714 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-23T14:35:19.714 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-23T14:35:19.714 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-03-23T14:35:19.715 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-03-23T14:35:19.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.854+0000 7f080176c640 1 Processor -- start 2024-03-23T14:35:19.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.854+0000 7f080176c640 1 -- start start 2024-03-23T14:35:19.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f080176c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07fc15e240 0x7f07fc06c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.852 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f080176c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc0ce4f0 0x7f07fc0ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f080176c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07fc0ceea0 con 0x7f07fc06cb80 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f080176c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f07fc06f2d0 con 0x7f07fc15e240 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f080176c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f07fc0cf0d0 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f07faffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07fc15e240 0x7f07fc06c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f07faffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f07fc15e240 0x7f07fc06c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41342/0 (socket says 172.21.15.161:41342) 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f07faffd640 1 -- 172.21.15.161:0/1079647456 learned_addr learned my addr 172.21.15.161:0/1079647456 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f07fb7fe640 1 --2- 172.21.15.161:0/1079647456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc0ce4f0 0x7f07fc0ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f07faffd640 1 -- 172.21.15.161:0/1079647456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc0ce4f0 msgr2=0x7f07fc0ce960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f07faffd640 1 --2- 172.21.15.161:0/1079647456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc0ce4f0 0x7f07fc0ce960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f07faffd640 1 -- 172.21.15.161:0/1079647456 >> v1:172.21.15.183:6789/0 conn(0x7f07fc06cb80 legacy=0x7f07fc0d08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.855+0000 7f07faffd640 1 -- 172.21.15.161:0/1079647456 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07fc0cf3b0 con 0x7f07fc15e240 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.856+0000 7f07faffd640 1 --2- 172.21.15.161:0/1079647456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc15e240 0x7f07fc06c640 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f07ec009e50 tx=0x7f07ec02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f09698ec9bd11d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.856+0000 7f07f9ffb640 1 -- 172.21.15.161:0/1079647456 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec02cb00 con 0x7f07fc15e240 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.856+0000 7f07f9ffb640 1 -- 172.21.15.161:0/1079647456 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07ec02cc60 con 0x7f07fc15e240 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.856+0000 7f07f9ffb640 1 -- 172.21.15.161:0/1079647456 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07ec0343f0 con 0x7f07fc15e240 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.857+0000 7f080176c640 1 -- 172.21.15.161:0/1079647456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc15e240 msgr2=0x7f07fc06c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.857+0000 7f080176c640 1 --2- 172.21.15.161:0/1079647456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc15e240 0x7f07fc06c640 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f07ec009e50 tx=0x7f07ec02bf60 comp rx=0 tx=0).stop 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.857+0000 7f080176c640 1 -- 172.21.15.161:0/1079647456 shutdown_connections 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.857+0000 7f080176c640 1 --2- 172.21.15.161:0/1079647456 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc0ce4f0 0x7f07fc0ce960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.857+0000 7f080176c640 1 --2- 172.21.15.161:0/1079647456 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc15e240 0x7f07fc06c640 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.857+0000 7f080176c640 1 -- 172.21.15.161:0/1079647456 >> 172.21.15.161:0/1079647456 conn(0x7f07fc09aa40 msgr2=0x7f07fc09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.857+0000 7f080176c640 1 -- 172.21.15.161:0/1079647456 shutdown_connections 2024-03-23T14:35:19.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.857+0000 7f080176c640 1 -- 172.21.15.161:0/1079647456 wait complete. 2024-03-23T14:35:19.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.858+0000 7f080176c640 1 Processor -- start 2024-03-23T14:35:19.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.858+0000 7f080176c640 1 -- start start 2024-03-23T14:35:19.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f080176c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc06cb80 0x7f07fc0ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f080176c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc0ce4f0 0x7f07fc15fc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f080176c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07fc15e240 0x7f07fc150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f080176c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc0ca710 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f080176c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f07fc1601b0 con 0x7f07fc15e240 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f080176c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f07fc0ca920 con 0x7f07fc06cb80 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc0ce4f0 0x7f07fc15fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc0ce4f0 0x7f07fc15fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41348/0 (socket says 172.21.15.161:41348) 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 -- 172.21.15.161:0/3265652532 learned_addr learned my addr 172.21.15.161:0/3265652532 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07faffd640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc06cb80 0x7f07fc0ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fb7fe640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07fc15e240 0x7f07fc150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 -- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc06cb80 msgr2=0x7f07fc0ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc06cb80 0x7f07fc0ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 -- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07fc15e240 msgr2=0x7f07fc150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:19.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07fc15e240 0x7f07fc150140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 -- 172.21.15.161:0/3265652532 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07ec009b30 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07fa7fc640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc0ce4f0 0x7f07fc15fc70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07f000c400 tx=0x7f07f000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07e37fe640 1 -- 172.21.15.161:0/3265652532 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f00052d0 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07e37fe640 1 -- 172.21.15.161:0/3265652532 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07f0005430 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07fc0cac00 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f07e37fe640 1 -- 172.21.15.161:0/3265652532 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07f0003830 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.859+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07fc0cb1c0 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.860+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07fc09fd60 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.861+0000 7f07e37fe640 1 -- 172.21.15.161:0/3265652532 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f07f0003990 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.861+0000 7f07e37fe640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f07e806b810 0x7f07e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:19.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.861+0000 7f07e37fe640 1 -- 172.21.15.161:0/3265652532 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f07f0089650 con 0x7f07fc0ce4f0 2024-03-23T14:35:19.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.861+0000 7f07faffd640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f07e806b810 0x7f07e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:19.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.862+0000 7f07faffd640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f07e806b810 0x7f07e806dcd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f07ec035000 tx=0x7f07ec030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:19.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:19.865+0000 7f07e37fe640 1 -- 172.21.15.161:0/3265652532 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f07f0054120 con 0x7f07fc0ce4f0 2024-03-23T14:35:20.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.072+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f07fc04f550 con 0x7f07fc0ce4f0 2024-03-23T14:35:20.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.073+0000 7f07e37fe640 1 -- 172.21.15.161:0/3265652532 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f07f00577e0 con 0x7f07fc0ce4f0 2024-03-23T14:35:20.068 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:20.069 INFO:teuthology.orchestra.run.smithi161.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f07e806b810 msgr2=0x7f07e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f07e806b810 0x7f07e806dcd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f07ec035000 tx=0x7f07ec030000 comp rx=0 tx=0).stop 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc0ce4f0 msgr2=0x7f07fc15fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc0ce4f0 0x7f07fc15fc70 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f07f000c400 tx=0x7f07f000c8c0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 shutdown_connections 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f07e806b810 0x7f07e806dcd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07fc15e240 0x7f07fc150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07fc0ce4f0 0x7f07fc15fc70 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 --2- 172.21.15.161:0/3265652532 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07fc06cb80 0x7f07fc0ca1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 >> 172.21.15.161:0/3265652532 conn(0x7f07fc09aa40 msgr2=0x7f07fc0d23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 shutdown_connections 2024-03-23T14:35:20.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.075+0000 7f080176c640 1 -- 172.21.15.161:0/3265652532 wait complete. 2024-03-23T14:35:20.081 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-03-23T14:35:20.082 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:20.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.186+0000 7f96bf82a640 1 Processor -- start 2024-03-23T14:35:20.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.186+0000 7f96bf82a640 1 -- start start 2024-03-23T14:35:20.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bf82a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b815e400 0x7f96b80cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bf82a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f96b80cf9d0 0x7f96b80d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bf82a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f96b80d1470 0x7f96b80d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bf82a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f96b80d1050 con 0x7f96b80cf9d0 2024-03-23T14:35:20.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bf82a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f96b80d0460 con 0x7f96b80d1470 2024-03-23T14:35:20.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bf82a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f96b80d06b0 con 0x7f96b815e400 2024-03-23T14:35:20.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bcdc5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f96b80cf9d0 0x7f96b80d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bddc7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f96b80d1470 0x7f96b80d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bcdc5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f96b80cf9d0 0x7f96b80d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41360/0 (socket says 172.21.15.161:41360) 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bcdc5640 1 -- 172.21.15.161:0/1396682076 learned_addr learned my addr 172.21.15.161:0/1396682076 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bd5c6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b815e400 0x7f96b80cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bcdc5640 1 -- 172.21.15.161:0/1396682076 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b815e400 msgr2=0x7f96b80cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bcdc5640 1 --2- 172.21.15.161:0/1396682076 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b815e400 0x7f96b80cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bcdc5640 1 -- 172.21.15.161:0/1396682076 >> v2:172.21.15.183:3300/0 conn(0x7f96b80d1470 msgr2=0x7f96b80d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bcdc5640 1 --2- 172.21.15.161:0/1396682076 >> v2:172.21.15.183:3300/0 conn(0x7f96b80d1470 0x7f96b80d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bcdc5640 1 -- 172.21.15.161:0/1396682076 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b80d0990 con 0x7f96b80cf9d0 2024-03-23T14:35:20.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.187+0000 7f96bddc7640 1 --2- 172.21.15.161:0/1396682076 >> v2:172.21.15.183:3300/0 conn(0x7f96b80d1470 0x7f96b80d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:20.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.188+0000 7f96bcdc5640 1 --2- 172.21.15.161:0/1396682076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80d2430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f96b808e3f0 tx=0x7f96a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef1565da8f520809 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.188+0000 7f96affff640 1 -- 172.21.15.161:0/1396682076 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8031870 con 0x7f96b80cf9d0 2024-03-23T14:35:20.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.188+0000 7f96affff640 1 -- 172.21.15.161:0/1396682076 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96a80319d0 con 0x7f96b80cf9d0 2024-03-23T14:35:20.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.188+0000 7f96affff640 1 -- 172.21.15.161:0/1396682076 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96a8031d60 con 0x7f96b80cf9d0 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 -- 172.21.15.161:0/1396682076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 msgr2=0x7f96b80d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 --2- 172.21.15.161:0/1396682076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80d2430 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f96b808e3f0 tx=0x7f96a802b910 comp rx=0 tx=0).stop 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 -- 172.21.15.161:0/1396682076 shutdown_connections 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 --2- 172.21.15.161:0/1396682076 >> v2:172.21.15.183:3300/0 conn(0x7f96b80d1470 0x7f96b80d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 --2- 172.21.15.161:0/1396682076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80d2430 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 --2- 172.21.15.161:0/1396682076 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b815e400 0x7f96b80cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 -- 172.21.15.161:0/1396682076 >> 172.21.15.161:0/1396682076 conn(0x7f96b809d190 msgr2=0x7f96b80d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 -- 172.21.15.161:0/1396682076 shutdown_connections 2024-03-23T14:35:20.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.189+0000 7f96bf82a640 1 -- 172.21.15.161:0/1396682076 wait complete. 2024-03-23T14:35:20.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.190+0000 7f96bf82a640 1 Processor -- start 2024-03-23T14:35:20.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.190+0000 7f96bf82a640 1 -- start start 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bf82a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80cdd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bf82a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b80d1470 0x7f96b8158e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bf82a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96b815e400 0x7f96b80c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bf82a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96b80b7b40 con 0x7f96b80cf9d0 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bf82a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f96b815cec0 con 0x7f96b815e400 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bf82a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f96b80ce2c0 con 0x7f96b80d1470 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80cdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80cdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41374/0 (socket says 172.21.15.161:41374) 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 -- 172.21.15.161:0/3138619030 learned_addr learned my addr 172.21.15.161:0/3138619030 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bddc7640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96b815e400 0x7f96b80c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bcdc5640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b80d1470 0x7f96b8158e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 -- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b80d1470 msgr2=0x7f96b8158e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b80d1470 0x7f96b8158e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 -- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96b815e400 msgr2=0x7f96b80c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96b815e400 0x7f96b80c9b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 -- 172.21.15.161:0/3138619030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96b0009a70 con 0x7f96b80cf9d0 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bddc7640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96b815e400 0x7f96b80c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96bd5c6640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80cdd80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f96b400e870 tx=0x7f96b400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.191+0000 7f96adffb640 1 -- 172.21.15.161:0/3138619030 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b40094f0 con 0x7f96b80cf9d0 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.192+0000 7f96adffb640 1 -- 172.21.15.161:0/3138619030 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96b400ee60 con 0x7f96b80cf9d0 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.192+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96a80094d0 con 0x7f96b80cf9d0 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.192+0000 7f96adffb640 1 -- 172.21.15.161:0/3138619030 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96b400cb80 con 0x7f96b80cf9d0 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.192+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96b80b72a0 con 0x7f96b80cf9d0 2024-03-23T14:35:20.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.193+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96b806d720 con 0x7f96b80cf9d0 2024-03-23T14:35:20.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.193+0000 7f96adffb640 1 -- 172.21.15.161:0/3138619030 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f96b400cce0 con 0x7f96b80cf9d0 2024-03-23T14:35:20.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.194+0000 7f96adffb640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f969c06b840 0x7f969c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.194+0000 7f96adffb640 1 -- 172.21.15.161:0/3138619030 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f96b408a140 con 0x7f96b80cf9d0 2024-03-23T14:35:20.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.194+0000 7f96bcdc5640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f969c06b840 0x7f969c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.194+0000 7f96bcdc5640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f969c06b840 0x7f969c06dd00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f96a8036000 tx=0x7f96a803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.198+0000 7f96adffb640 1 -- 172.21.15.161:0/3138619030 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f96b4054c10 con 0x7f96b80cf9d0 2024-03-23T14:35:20.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.371+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f96b80a1e00 con 0x7f96b80cf9d0 2024-03-23T14:35:20.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.372+0000 7f96adffb640 1 -- 172.21.15.161:0/3138619030 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f96b40582d0 con 0x7f96b80cf9d0 2024-03-23T14:35:20.368 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:20.369 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f969c06b840 msgr2=0x7f969c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f969c06b840 0x7f969c06dd00 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f96a8036000 tx=0x7f96a803c040 comp rx=0 tx=0).stop 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 msgr2=0x7f96b80cdd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80cdd80 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f96b400e870 tx=0x7f96b400ec70 comp rx=0 tx=0).stop 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 shutdown_connections 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f969c06b840 0x7f969c06dd00 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96b815e400 0x7f96b80c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96b80d1470 0x7f96b8158e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.374+0000 7f96bf82a640 1 --2- 172.21.15.161:0/3138619030 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96b80cf9d0 0x7f96b80cdd80 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.375+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 >> 172.21.15.161:0/3138619030 conn(0x7f96b809d190 msgr2=0x7f96b80d3df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:20.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.375+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 shutdown_connections 2024-03-23T14:35:20.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.375+0000 7f96bf82a640 1 -- 172.21.15.161:0/3138619030 wait complete. 2024-03-23T14:35:20.381 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:20.382 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.490+0000 7f37ebff4640 1 Processor -- start 2024-03-23T14:35:20.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.490+0000 7f37ebff4640 1 -- start start 2024-03-23T14:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37ebff4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37e415dfb0 0x7f37e40d1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37ebff4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e415e390 0x7f37e40d2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37ebff4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f37e40d0a40 0x7f37e40d0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37ebff4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f37e40cc4f0 con 0x7f37e415dfb0 2024-03-23T14:35:20.488 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37ebff4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f37e40ccab0 con 0x7f37e40d0a40 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37ebff4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f37e40d1770 con 0x7f37e415e390 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37e9d90640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37e415dfb0 0x7f37e40d1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37ea591640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f37e40d0a40 0x7f37e40d0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37e958f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e415e390 0x7f37e40d2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37e9d90640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f37e415dfb0 0x7f37e40d1e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41390/0 (socket says 172.21.15.161:41390) 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37e9d90640 1 -- 172.21.15.161:0/2908420207 learned_addr learned my addr 172.21.15.161:0/2908420207 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37e9d90640 1 -- 172.21.15.161:0/2908420207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e415e390 msgr2=0x7f37e40d2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37e9d90640 1 --2- 172.21.15.161:0/2908420207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e415e390 0x7f37e40d2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37e9d90640 1 -- 172.21.15.161:0/2908420207 >> v2:172.21.15.183:3300/0 conn(0x7f37e40d0a40 msgr2=0x7f37e40d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.491+0000 7f37e9d90640 1 --2- 172.21.15.161:0/2908420207 >> v2:172.21.15.183:3300/0 conn(0x7f37e40d0a40 0x7f37e40d0ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.489 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.492+0000 7f37e9d90640 1 -- 172.21.15.161:0/2908420207 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37e40d1a50 con 0x7f37e415dfb0 2024-03-23T14:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.492+0000 7f37e9d90640 1 --2- 172.21.15.161:0/2908420207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e40d1e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37e40ce4e0 tx=0x7f37d402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ab9fafe85337831 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.492+0000 7f37e8d8e640 1 -- 172.21.15.161:0/2908420207 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4031870 con 0x7f37e415dfb0 2024-03-23T14:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.492+0000 7f37e8d8e640 1 -- 172.21.15.161:0/2908420207 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37d40319d0 con 0x7f37e415dfb0 2024-03-23T14:35:20.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.492+0000 7f37e8d8e640 1 -- 172.21.15.161:0/2908420207 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37d4031d60 con 0x7f37e415dfb0 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 -- 172.21.15.161:0/2908420207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 msgr2=0x7f37e40d1e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 --2- 172.21.15.161:0/2908420207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e40d1e10 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f37e40ce4e0 tx=0x7f37d402b910 comp rx=0 tx=0).stop 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 -- 172.21.15.161:0/2908420207 shutdown_connections 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 --2- 172.21.15.161:0/2908420207 >> v2:172.21.15.183:3300/0 conn(0x7f37e40d0a40 0x7f37e40d0ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 --2- 172.21.15.161:0/2908420207 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e415e390 0x7f37e40d2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 --2- 172.21.15.161:0/2908420207 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e40d1e10 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 -- 172.21.15.161:0/2908420207 >> 172.21.15.161:0/2908420207 conn(0x7f37e409aa40 msgr2=0x7f37e409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 -- 172.21.15.161:0/2908420207 shutdown_connections 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.493+0000 7f37ebff4640 1 -- 172.21.15.161:0/2908420207 wait complete. 2024-03-23T14:35:20.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.494+0000 7f37ebff4640 1 Processor -- start 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.494+0000 7f37ebff4640 1 -- start start 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37ebff4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e40d0a40 0x7f37e4151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37ebff4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e41f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37ebff4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37e415e390 0x7f37e415fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37ebff4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f37e4152480 con 0x7f37e415dfb0 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37ebff4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f37e41f7500 con 0x7f37e415e390 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37ebff4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f37e414e0d0 con 0x7f37e40d0a40 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e41f6fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e41f6fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41404/0 (socket says 172.21.15.161:41404) 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 -- 172.21.15.161:0/595114305 learned_addr learned my addr 172.21.15.161:0/595114305 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:20.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37ea591640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37e415e390 0x7f37e415fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e9d90640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e40d0a40 0x7f37e4151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 -- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e40d0a40 msgr2=0x7f37e4151b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e40d0a40 0x7f37e4151b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 -- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37e415e390 msgr2=0x7f37e415fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37e415e390 0x7f37e415fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 -- 172.21.15.161:0/595114305 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37d40094d0 con 0x7f37e415dfb0 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.495+0000 7f37e958f640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e41f6fc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f37e000e840 tx=0x7f37e000ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.496+0000 7f37da7fc640 1 -- 172.21.15.161:0/595114305 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e00094f0 con 0x7f37e415dfb0 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.496+0000 7f37da7fc640 1 -- 172.21.15.161:0/595114305 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37e000c940 con 0x7f37e415dfb0 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.496+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37e41f7730 con 0x7f37e415dfb0 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.496+0000 7f37da7fc640 1 -- 172.21.15.161:0/595114305 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e000ccd0 con 0x7f37e415dfb0 2024-03-23T14:35:20.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.496+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37e41f7ac0 con 0x7f37e415dfb0 2024-03-23T14:35:20.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.497+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37e404f4e0 con 0x7f37e415dfb0 2024-03-23T14:35:20.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.497+0000 7f37da7fc640 1 -- 172.21.15.161:0/595114305 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f37e0013470 con 0x7f37e415dfb0 2024-03-23T14:35:20.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.497+0000 7f37da7fc640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f37c806b810 0x7f37c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.497+0000 7f37da7fc640 1 -- 172.21.15.161:0/595114305 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f37e008a150 con 0x7f37e415dfb0 2024-03-23T14:35:20.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.497+0000 7f37e9d90640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f37c806b810 0x7f37c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.498+0000 7f37e9d90640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f37c806b810 0x7f37c806dcd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f37d4036000 tx=0x7f37d403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.501+0000 7f37da7fc640 1 -- 172.21.15.161:0/595114305 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f37e0054cb0 con 0x7f37e415dfb0 2024-03-23T14:35:20.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.675+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f37e40685f0 con 0x7f37e415dfb0 2024-03-23T14:35:20.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.675+0000 7f37da7fc640 1 -- 172.21.15.161:0/595114305 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f37e0058370 con 0x7f37e415dfb0 2024-03-23T14:35:20.671 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.677+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f37c806b810 msgr2=0x7f37c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.677+0000 7f37ebff4640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f37c806b810 0x7f37c806dcd0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f37d4036000 tx=0x7f37d403c040 comp rx=0 tx=0).stop 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.677+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 msgr2=0x7f37e41f6fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.677+0000 7f37ebff4640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e41f6fc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f37e000e840 tx=0x7f37e000ed00 comp rx=0 tx=0).stop 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.678+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 shutdown_connections 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.678+0000 7f37ebff4640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f37c806b810 0x7f37c806dcd0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.678+0000 7f37ebff4640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f37e415e390 0x7f37e415fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.678+0000 7f37ebff4640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f37e415dfb0 0x7f37e41f6fc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.678+0000 7f37ebff4640 1 --2- 172.21.15.161:0/595114305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f37e40d0a40 0x7f37e4151b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.678+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 >> 172.21.15.161:0/595114305 conn(0x7f37e409aa40 msgr2=0x7f37e406ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.678+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 shutdown_connections 2024-03-23T14:35:20.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.678+0000 7f37ebff4640 1 -- 172.21.15.161:0/595114305 wait complete. 2024-03-23T14:35:20.684 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:20.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.794+0000 7f8b4c377640 1 Processor -- start 2024-03-23T14:35:20.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.794+0000 7f8b4c377640 1 -- start start 2024-03-23T14:35:20.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4c377640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8b441615a0 0x7f8b440ad1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4c377640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b440b16f0 0x7f8b440cf4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4c377640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b440ad6f0 con 0x7f8b44161980 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4c377640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8b440afd80 con 0x7f8b440b16f0 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4c377640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f8b44162d60 con 0x7f8b441615a0 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4a914640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b440b16f0 0x7f8b440cf4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4a914640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b440b16f0 0x7f8b440cf4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33258/0 (socket says 172.21.15.161:33258) 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4a914640 1 -- 172.21.15.161:0/2083083802 learned_addr learned my addr 172.21.15.161:0/2083083802 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.795+0000 7f8b4a113640 1 --2- 172.21.15.161:0/2083083802 >> v2:172.21.15.183:3300/0 conn(0x7f8b441615a0 0x7f8b440ad1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.796+0000 7f8b4a914640 1 -- 172.21.15.161:0/2083083802 >> v2:172.21.15.183:3300/0 conn(0x7f8b441615a0 msgr2=0x7f8b440ad1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.796+0000 7f8b4a914640 1 --2- 172.21.15.161:0/2083083802 >> v2:172.21.15.183:3300/0 conn(0x7f8b441615a0 0x7f8b440ad1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.796+0000 7f8b4a914640 1 -- 172.21.15.161:0/2083083802 >> v1:172.21.15.183:6789/0 conn(0x7f8b44161980 legacy=0x7f8b440b0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.796+0000 7f8b4a914640 1 -- 172.21.15.161:0/2083083802 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b440b1ba0 con 0x7f8b440b16f0 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.796+0000 7f8b4a914640 1 --2- 172.21.15.161:0/2083083802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b440b16f0 0x7f8b440cf4d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b44070cd0 tx=0x7f8b3c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1604b10ac3f6755 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.796+0000 7f8b49111640 1 -- 172.21.15.161:0/2083083802 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c031870 con 0x7f8b440b16f0 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.796+0000 7f8b49111640 1 -- 172.21.15.161:0/2083083802 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b3c0319d0 con 0x7f8b440b16f0 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.796+0000 7f8b49111640 1 -- 172.21.15.161:0/2083083802 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b3c031d60 con 0x7f8b440b16f0 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.797+0000 7f8b4c377640 1 -- 172.21.15.161:0/2083083802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b440b16f0 msgr2=0x7f8b440cf4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.797+0000 7f8b4c377640 1 --2- 172.21.15.161:0/2083083802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b440b16f0 0x7f8b440cf4d0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8b44070cd0 tx=0x7f8b3c02b910 comp rx=0 tx=0).stop 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.797+0000 7f8b4c377640 1 -- 172.21.15.161:0/2083083802 shutdown_connections 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.797+0000 7f8b4c377640 1 --2- 172.21.15.161:0/2083083802 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b440b16f0 0x7f8b440cf4d0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.797+0000 7f8b4c377640 1 --2- 172.21.15.161:0/2083083802 >> v2:172.21.15.183:3300/0 conn(0x7f8b441615a0 0x7f8b440ad1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.797+0000 7f8b4c377640 1 -- 172.21.15.161:0/2083083802 >> 172.21.15.161:0/2083083802 conn(0x7f8b440b1ed0 msgr2=0x7f8b440b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:20.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.797+0000 7f8b4c377640 1 -- 172.21.15.161:0/2083083802 shutdown_connections 2024-03-23T14:35:20.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.797+0000 7f8b4c377640 1 -- 172.21.15.161:0/2083083802 wait complete. 2024-03-23T14:35:20.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.798+0000 7f8b4c377640 1 Processor -- start 2024-03-23T14:35:20.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.798+0000 7f8b4c377640 1 -- start start 2024-03-23T14:35:20.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b4c377640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b440b16f0 0x7f8b44154fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b4c377640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b441615a0 0x7f8b440ca9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b4c377640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b44161980 0x7f8b440caf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b4c377640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b44155520 con 0x7f8b441615a0 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b4c377640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8b440cd820 con 0x7f8b44161980 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b4c377640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8b440cba80 con 0x7f8b440b16f0 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b441615a0 0x7f8b440ca9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b441615a0 0x7f8b440ca9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33272/0 (socket says 172.21.15.161:33272) 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 -- 172.21.15.161:0/3942421064 learned_addr learned my addr 172.21.15.161:0/3942421064 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b4a914640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b44161980 0x7f8b440caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b4a113640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b440b16f0 0x7f8b44154fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 -- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b440b16f0 msgr2=0x7f8b44154fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b440b16f0 0x7f8b44154fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 -- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b44161980 msgr2=0x7f8b440caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b44161980 0x7f8b440caf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 -- 172.21.15.161:0/3942421064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b3c0094d0 con 0x7f8b441615a0 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b49912640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b441615a0 0x7f8b440ca9d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c3c0 tx=0x7f8b3400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.799+0000 7f8b3affd640 1 -- 172.21.15.161:0/3942421064 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b340052d0 con 0x7f8b441615a0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.800+0000 7f8b3affd640 1 -- 172.21.15.161:0/3942421064 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b34005450 con 0x7f8b441615a0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.800+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b440cbd60 con 0x7f8b441615a0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.800+0000 7f8b3affd640 1 -- 172.21.15.161:0/3942421064 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b34007260 con 0x7f8b441615a0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.800+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b441559a0 con 0x7f8b441615a0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.801+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b440ad1b0 con 0x7f8b441615a0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.801+0000 7f8b3affd640 1 -- 172.21.15.161:0/3942421064 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8b340073c0 con 0x7f8b441615a0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.801+0000 7f8b3affd640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b2806b810 0x7f8b2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.801+0000 7f8b3affd640 1 -- 172.21.15.161:0/3942421064 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8b340894a0 con 0x7f8b441615a0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.801+0000 7f8b4a113640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b2806b810 0x7f8b2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:20.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.802+0000 7f8b4a113640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b2806b810 0x7f8b2806dcd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8b400113e0 tx=0x7f8b4000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:20.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.806+0000 7f8b3affd640 1 -- 172.21.15.161:0/3942421064 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b34053f70 con 0x7f8b441615a0 2024-03-23T14:35:20.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.978+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8b4409e530 con 0x7f8b441615a0 2024-03-23T14:35:20.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.979+0000 7f8b3affd640 1 -- 172.21.15.161:0/3942421064 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f8b34057630 con 0x7f8b441615a0 2024-03-23T14:35:20.975 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b2806b810 msgr2=0x7f8b2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b2806b810 0x7f8b2806dcd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f8b400113e0 tx=0x7f8b4000b040 comp rx=0 tx=0).stop 2024-03-23T14:35:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b441615a0 msgr2=0x7f8b440ca9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b441615a0 0x7f8b440ca9d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f8b3400c3c0 tx=0x7f8b3400c880 comp rx=0 tx=0).stop 2024-03-23T14:35:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 shutdown_connections 2024-03-23T14:35:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b2806b810 0x7f8b2806dcd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b44161980 0x7f8b440caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b441615a0 0x7f8b440ca9d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 --2- 172.21.15.161:0/3942421064 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b440b16f0 0x7f8b44154fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:20.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 >> 172.21.15.161:0/3942421064 conn(0x7f8b440b1ed0 msgr2=0x7f8b440adfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:20.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 shutdown_connections 2024-03-23T14:35:20.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:20.981+0000 7f8b4c377640 1 -- 172.21.15.161:0/3942421064 wait complete. 2024-03-23T14:35:20.987 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.094+0000 7fb10a5e2640 1 Processor -- start 2024-03-23T14:35:21.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.094+0000 7fb10a5e2640 1 -- start start 2024-03-23T14:35:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.095+0000 7fb10a5e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e850 0x7fb1040cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.095+0000 7fb10a5e2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1040d4ce0 con 0x7fb10415e470 2024-03-23T14:35:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.095+0000 7fb10a5e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1040cf800 con 0x7fb1040d01d0 2024-03-23T14:35:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.095+0000 7fb10a5e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb1040ceed0 con 0x7fb10415e850 2024-03-23T14:35:21.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.095+0000 7fb103fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb10415e470 0x7fb1040d45d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57188/0 (socket says 172.21.15.161:57188) 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.095+0000 7fb103fff640 1 -- 172.21.15.161:0/1478693307 learned_addr learned my addr 172.21.15.161:0/1478693307 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.095+0000 7fb1037fe640 1 --2- 172.21.15.161:0/1478693307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e850 0x7fb1040cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.096+0000 7fb102ffd640 1 -- 172.21.15.161:0/1478693307 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 109962325 0 0) 0x7fb0f4001110 con 0x7fb10415e470 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.096+0000 7fb102ffd640 1 -- 172.21.15.161:0/1478693307 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0ec003520 con 0x7fb10415e470 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.096+0000 7fb102ffd640 1 -- 172.21.15.161:0/1478693307 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1306865978 0 0) 0x7fb1040cf800 con 0x7fb1040d01d0 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.096+0000 7fb102ffd640 1 -- 172.21.15.161:0/1478693307 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb0f4001110 con 0x7fb1040d01d0 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.096+0000 7fb1037fe640 1 -- 172.21.15.161:0/1478693307 >> v1:172.21.15.183:6789/0 conn(0x7fb1040d01d0 legacy=0x7fb1040d24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.096+0000 7fb1037fe640 1 -- 172.21.15.161:0/1478693307 >> v1:172.21.15.161:6789/0 conn(0x7fb10415e470 legacy=0x7fb1040d45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.096+0000 7fb1037fe640 1 -- 172.21.15.161:0/1478693307 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1040cf1b0 con 0x7fb10415e850 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.096+0000 7fb1037fe640 1 --2- 172.21.15.161:0/1478693307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e850 0x7fb1040cfc90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80097d0 tx=0x7fb0f802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5bce7e45b44fb701 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb102ffd640 1 -- 172.21.15.161:0/1478693307 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8031800 con 0x7fb10415e850 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb102ffd640 1 -- 172.21.15.161:0/1478693307 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f8031960 con 0x7fb10415e850 2024-03-23T14:35:21.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb102ffd640 1 -- 172.21.15.161:0/1478693307 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8031cf0 con 0x7fb10415e850 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb10a5e2640 1 -- 172.21.15.161:0/1478693307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e850 msgr2=0x7fb1040cfc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb10a5e2640 1 --2- 172.21.15.161:0/1478693307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e850 0x7fb1040cfc90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fb0f80097d0 tx=0x7fb0f802b8a0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb10a5e2640 1 -- 172.21.15.161:0/1478693307 shutdown_connections 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb10a5e2640 1 --2- 172.21.15.161:0/1478693307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e850 0x7fb1040cfc90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb10a5e2640 1 -- 172.21.15.161:0/1478693307 >> 172.21.15.161:0/1478693307 conn(0x7fb10409d180 msgr2=0x7fb1040d3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb10a5e2640 1 -- 172.21.15.161:0/1478693307 shutdown_connections 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.097+0000 7fb10a5e2640 1 -- 172.21.15.161:0/1478693307 wait complete. 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.098+0000 7fb10a5e2640 1 Processor -- start 2024-03-23T14:35:21.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.098+0000 7fb10a5e2640 1 -- start start 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb10a5e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e470 0x7fb10406d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb10a5e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb10415e850 0x7fb10415b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb10a5e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1040c7340 0x7fb1040c9a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb10a5e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb10406d7e0 con 0x7fb10415e850 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb10a5e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb1040b9850 con 0x7fb1040c7340 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb10a5e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb10415b880 con 0x7fb10415e470 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb1037fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb10415e850 0x7fb10415b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb1037fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb10415e850 0x7fb10415b340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33276/0 (socket says 172.21.15.161:33276) 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb1037fe640 1 -- 172.21.15.161:0/2329804317 learned_addr learned my addr 172.21.15.161:0/2329804317 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb108b7f640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1040c7340 0x7fb1040c9a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb103fff640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e470 0x7fb10406d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.099+0000 7fb1037fe640 1 -- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e470 msgr2=0x7fb10406d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb1037fe640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e470 0x7fb10406d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb1037fe640 1 -- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1040c7340 msgr2=0x7fb1040c9a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb1037fe640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1040c7340 0x7fb1040c9a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb1037fe640 1 -- 172.21.15.161:0/2329804317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f8009400 con 0x7fb10415e850 2024-03-23T14:35:21.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb1037fe640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb10415e850 0x7fb10415b340 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8034040 tx=0x7fb0f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb100ff9640 1 -- 172.21.15.161:0/2329804317 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f803ac60 con 0x7fb10415e850 2024-03-23T14:35:21.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb100ff9640 1 -- 172.21.15.161:0/2329804317 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0f803ae00 con 0x7fb10415e850 2024-03-23T14:35:21.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb100ff9640 1 -- 172.21.15.161:0/2329804317 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0f8044630 con 0x7fb10415e850 2024-03-23T14:35:21.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1040b8c40 con 0x7fb10415e850 2024-03-23T14:35:21.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.100+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1040b9200 con 0x7fb10415e850 2024-03-23T14:35:21.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.101+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb10404f4e0 con 0x7fb10415e850 2024-03-23T14:35:21.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.101+0000 7fb100ff9640 1 -- 172.21.15.161:0/2329804317 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb0f804d450 con 0x7fb10415e850 2024-03-23T14:35:21.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.102+0000 7fb100ff9640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb0f006b810 0x7fb0f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.102+0000 7fb100ff9640 1 -- 172.21.15.161:0/2329804317 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb0f802e070 con 0x7fb10415e850 2024-03-23T14:35:21.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.102+0000 7fb103fff640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb0f006b810 0x7fb0f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.102+0000 7fb103fff640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb0f006b810 0x7fb0f006dcd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4000f60 tx=0x7fb0f4007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.106+0000 7fb100ff9640 1 -- 172.21.15.161:0/2329804317 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb0f8076d50 con 0x7fb10415e850 2024-03-23T14:35:21.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.278+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb10415cb20 con 0x7fb10415e850 2024-03-23T14:35:21.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.278+0000 7fb100ff9640 1 -- 172.21.15.161:0/2329804317 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fb0f8006e10 con 0x7fb10415e850 2024-03-23T14:35:21.275 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:21.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.280+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb0f006b810 msgr2=0x7fb0f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.280+0000 7fb10a5e2640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb0f006b810 0x7fb0f006dcd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fb0f4000f60 tx=0x7fb0f4007d50 comp rx=0 tx=0).stop 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.280+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb10415e850 msgr2=0x7fb10415b340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.280+0000 7fb10a5e2640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb10415e850 0x7fb10415b340 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb0f8034040 tx=0x7fb0f80046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.281+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 shutdown_connections 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.281+0000 7fb10a5e2640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb0f006b810 0x7fb0f006dcd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.281+0000 7fb10a5e2640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1040c7340 0x7fb1040c9a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.281+0000 7fb10a5e2640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb10415e850 0x7fb10415b340 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.281+0000 7fb10a5e2640 1 --2- 172.21.15.161:0/2329804317 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb10415e470 0x7fb10406d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.281+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 >> 172.21.15.161:0/2329804317 conn(0x7fb10409d180 msgr2=0x7fb1040d1dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.281+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 shutdown_connections 2024-03-23T14:35:21.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.281+0000 7fb10a5e2640 1 -- 172.21.15.161:0/2329804317 wait complete. 2024-03-23T14:35:21.287 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:21.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.392+0000 7fda4e0eb640 1 Processor -- start 2024-03-23T14:35:21.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.392+0000 7fda4e0eb640 1 -- start start 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda4e0eb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fda48072100 0x7fda4815b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda4e0eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda480724e0 0x7fda4806e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda4e0eb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda4815be70 con 0x7fda4815b930 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda4e0eb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fda4815c430 con 0x7fda480724e0 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda4e0eb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fda4815c150 con 0x7fda48072100 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda46ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda480724e0 0x7fda4806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda46ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda480724e0 0x7fda4806e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33292/0 (socket says 172.21.15.161:33292) 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda46ffd640 1 -- 172.21.15.161:0/2545192580 learned_addr learned my addr 172.21.15.161:0/2545192580 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:21.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda477fe640 1 --2- 172.21.15.161:0/2545192580 >> v2:172.21.15.183:3300/0 conn(0x7fda48072100 0x7fda4815b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda46ffd640 1 -- 172.21.15.161:0/2545192580 >> v2:172.21.15.183:3300/0 conn(0x7fda48072100 msgr2=0x7fda4815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda46ffd640 1 --2- 172.21.15.161:0/2545192580 >> v2:172.21.15.183:3300/0 conn(0x7fda48072100 0x7fda4815b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda467fc640 1 -- 172.21.15.161:0/2545192580 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 341545017 0 0) 0x7fda4815be70 con 0x7fda4815b930 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda46ffd640 1 -- 172.21.15.161:0/2545192580 >> v1:172.21.15.183:6789/0 conn(0x7fda4815b930 legacy=0x7fda4815e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.393+0000 7fda46ffd640 1 -- 172.21.15.161:0/2545192580 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda480b2c70 con 0x7fda480724e0 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.394+0000 7fda46ffd640 1 --2- 172.21.15.161:0/2545192580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda4806e590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fda4806cac0 tx=0x7fda3c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cbd0d7bec8eb5c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.394+0000 7fda467fc640 1 -- 172.21.15.161:0/2545192580 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c030860 con 0x7fda480724e0 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.394+0000 7fda467fc640 1 -- 172.21.15.161:0/2545192580 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3c0309c0 con 0x7fda480724e0 2024-03-23T14:35:21.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.394+0000 7fda467fc640 1 -- 172.21.15.161:0/2545192580 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c030d50 con 0x7fda480724e0 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.395+0000 7fda4e0eb640 1 -- 172.21.15.161:0/2545192580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 msgr2=0x7fda4806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.395+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/2545192580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda4806e590 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fda4806cac0 tx=0x7fda3c02b9a0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.395+0000 7fda4e0eb640 1 -- 172.21.15.161:0/2545192580 shutdown_connections 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.395+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/2545192580 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda4806e590 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.395+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/2545192580 >> v2:172.21.15.183:3300/0 conn(0x7fda48072100 0x7fda4815b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.395+0000 7fda4e0eb640 1 -- 172.21.15.161:0/2545192580 >> 172.21.15.161:0/2545192580 conn(0x7fda480b17f0 msgr2=0x7fda480b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.395+0000 7fda4e0eb640 1 -- 172.21.15.161:0/2545192580 shutdown_connections 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.395+0000 7fda4e0eb640 1 -- 172.21.15.161:0/2545192580 wait complete. 2024-03-23T14:35:21.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.396+0000 7fda4e0eb640 1 Processor -- start 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.396+0000 7fda4e0eb640 1 -- start start 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda4e0eb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda48072100 0x7fda481480d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda4e0eb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda481e6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda4e0eb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4815b930 0x7fda4814a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda4e0eb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda481e5ff0 con 0x7fda480724e0 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda4e0eb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fda4814b300 con 0x7fda48072100 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda4e0eb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fda481e5a00 con 0x7fda4815b930 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda481e6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda481e6410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33302/0 (socket says 172.21.15.161:33302) 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 -- 172.21.15.161:0/890057479 learned_addr learned my addr 172.21.15.161:0/890057479 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda477fe640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda48072100 0x7fda481480d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda47fff640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4815b930 0x7fda4814a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 -- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4815b930 msgr2=0x7fda4814a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4815b930 0x7fda4814a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 -- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda48072100 msgr2=0x7fda481480d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda48072100 0x7fda481480d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 -- 172.21.15.161:0/890057479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda3c0094d0 con 0x7fda480724e0 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.397+0000 7fda46ffd640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda481e6410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fda3c000c00 tx=0x7fda3c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.398+0000 7fda2ffff640 1 -- 172.21.15.161:0/890057479 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c039c90 con 0x7fda480724e0 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.398+0000 7fda2ffff640 1 -- 172.21.15.161:0/890057479 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda3c039df0 con 0x7fda480724e0 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.398+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda48152150 con 0x7fda480724e0 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.398+0000 7fda2ffff640 1 -- 172.21.15.161:0/890057479 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda3c043620 con 0x7fda480724e0 2024-03-23T14:35:21.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.398+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda4814ad40 con 0x7fda480724e0 2024-03-23T14:35:21.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.399+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda481e6130 con 0x7fda480724e0 2024-03-23T14:35:21.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.399+0000 7fda2ffff640 1 -- 172.21.15.161:0/890057479 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fda3c043780 con 0x7fda480724e0 2024-03-23T14:35:21.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.399+0000 7fda2ffff640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda3406b810 0x7fda3406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.399+0000 7fda2ffff640 1 -- 172.21.15.161:0/890057479 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fda3c03e070 con 0x7fda480724e0 2024-03-23T14:35:21.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.399+0000 7fda477fe640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda3406b810 0x7fda3406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.397 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.400+0000 7fda477fe640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda3406b810 0x7fda3406dcd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fda38009b40 tx=0x7fda38009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.404+0000 7fda2ffff640 1 -- 172.21.15.161:0/890057479 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fda481e6130 con 0x7fda480724e0 2024-03-23T14:35:21.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.615+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fda481e6130 con 0x7fda480724e0 2024-03-23T14:35:21.611 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.616+0000 7fda2ffff640 1 -- 172.21.15.161:0/890057479 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5181 (secure 0 0 0) 0x7fda481e6130 con 0x7fda480724e0 2024-03-23T14:35:21.612 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:21.612 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:35:13.112471+0000","modified":"2024-03-23T14:35:15.108804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-23T14:35:21.612 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 6 2024-03-23T14:35:21.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda3406b810 msgr2=0x7fda3406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda3406b810 0x7fda3406dcd0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fda38009b40 tx=0x7fda38009190 comp rx=0 tx=0).stop 2024-03-23T14:35:21.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 msgr2=0x7fda481e6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda481e6410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fda3c000c00 tx=0x7fda3c004730 comp rx=0 tx=0).stop 2024-03-23T14:35:21.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 shutdown_connections 2024-03-23T14:35:21.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda3406b810 0x7fda3406dcd0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda4815b930 0x7fda4814a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda480724e0 0x7fda481e6410 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 --2- 172.21.15.161:0/890057479 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda48072100 0x7fda481480d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 >> 172.21.15.161:0/890057479 conn(0x7fda480b17f0 msgr2=0x7fda480b3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:21.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 shutdown_connections 2024-03-23T14:35:21.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.618+0000 7fda4e0eb640 1 -- 172.21.15.161:0/890057479 wait complete. 2024-03-23T14:35:21.625 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:21.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.729+0000 7fd7270c3640 1 Processor -- start 2024-03-23T14:35:21.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.729+0000 7fd7270c3640 1 -- start start 2024-03-23T14:35:21.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd7270c3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd72015b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd7270c3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd72006eca0 0x7fd7200b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd7270c3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd72015be70 con 0x7fd7200724e0 2024-03-23T14:35:21.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd7270c3640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fd72015c430 con 0x7fd72006eca0 2024-03-23T14:35:21.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd7270c3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd72015c150 con 0x7fd720072100 2024-03-23T14:35:21.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd724e5f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd72015b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd725660640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd72006eca0 0x7fd7200b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd71ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd7200724e0 0x7fd72006e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:58686/0 (socket says 172.21.15.161:58686) 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd724e5f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd72015b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:55868/0 (socket says 172.21.15.161:55868) 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd725660640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd72006eca0 0x7fd7200b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:44854/0 (socket says 172.21.15.161:44854) 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd724e5f640 1 -- 172.21.15.161:0/2252143230 learned_addr learned my addr 172.21.15.161:0/2252143230 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd724e5f640 1 -- 172.21.15.161:0/2252143230 >> v2:172.21.15.183:3300/0 conn(0x7fd72006eca0 msgr2=0x7fd7200b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.730+0000 7fd724e5f640 1 --2- 172.21.15.161:0/2252143230 >> v2:172.21.15.183:3300/0 conn(0x7fd72006eca0 0x7fd7200b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.731+0000 7fd724e5f640 1 -- 172.21.15.161:0/2252143230 >> v1:172.21.15.183:6789/0 conn(0x7fd7200724e0 legacy=0x7fd72006e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.731+0000 7fd724e5f640 1 -- 172.21.15.161:0/2252143230 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7200b2c70 con 0x7fd720072100 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.731+0000 7fd724e5f640 1 --2- 172.21.15.161:0/2252143230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd72015b3f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd710009870 tx=0x7fd71002b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14b3b76c72cfe1c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.731+0000 7fd71f7fe640 1 -- 172.21.15.161:0/2252143230 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710030840 con 0x7fd720072100 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.731+0000 7fd71f7fe640 1 -- 172.21.15.161:0/2252143230 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7100309a0 con 0x7fd720072100 2024-03-23T14:35:21.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.731+0000 7fd71f7fe640 1 -- 172.21.15.161:0/2252143230 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd710030d30 con 0x7fd720072100 2024-03-23T14:35:21.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.732+0000 7fd7270c3640 1 -- 172.21.15.161:0/2252143230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 msgr2=0x7fd72015b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.732+0000 7fd7270c3640 1 --2- 172.21.15.161:0/2252143230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd72015b3f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd710009870 tx=0x7fd71002b980 comp rx=0 tx=0).stop 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.732+0000 7fd7270c3640 1 -- 172.21.15.161:0/2252143230 shutdown_connections 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.732+0000 7fd7270c3640 1 --2- 172.21.15.161:0/2252143230 >> v2:172.21.15.183:3300/0 conn(0x7fd72006eca0 0x7fd7200b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.732+0000 7fd7270c3640 1 --2- 172.21.15.161:0/2252143230 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd72015b3f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.732+0000 7fd7270c3640 1 -- 172.21.15.161:0/2252143230 >> 172.21.15.161:0/2252143230 conn(0x7fd7200b17f0 msgr2=0x7fd7200b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.732+0000 7fd7270c3640 1 -- 172.21.15.161:0/2252143230 shutdown_connections 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.732+0000 7fd7270c3640 1 -- 172.21.15.161:0/2252143230 wait complete. 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.733+0000 7fd7270c3640 1 Processor -- start 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.733+0000 7fd7270c3640 1 -- start start 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd7270c3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd72006eca0 0x7fd7201560f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd7270c3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd720142bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd7270c3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7200724e0 0x7fd7201430f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd7270c3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd720141460 con 0x7fd7200724e0 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd7270c3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd720141b10 con 0x7fd72006eca0 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd7270c3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd720141df0 con 0x7fd720072100 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd725660640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7200724e0 0x7fd7201430f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd725660640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7200724e0 0x7fd7201430f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33316/0 (socket says 172.21.15.161:33316) 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd725660640 1 -- 172.21.15.161:0/3434756197 learned_addr learned my addr 172.21.15.161:0/3434756197 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd724e5f640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd72006eca0 0x7fd7201560f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd71ffff640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd720142bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd725660640 1 -- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 msgr2=0x7fd720142bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd725660640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd720142bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.734+0000 7fd725660640 1 -- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd72006eca0 msgr2=0x7fd7201560f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd725660640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd72006eca0 0x7fd7201560f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd725660640 1 -- 172.21.15.161:0/3434756197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7100094d0 con 0x7fd7200724e0 2024-03-23T14:35:21.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd724e5f640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd72006eca0 0x7fd7201560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:21.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd725660640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7200724e0 0x7fd7201430f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd71800efc0 tx=0x7fd71800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd71d7fa640 1 -- 172.21.15.161:0/3434756197 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd718009b10 con 0x7fd7200724e0 2024-03-23T14:35:21.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd71d7fa640 1 -- 172.21.15.161:0/3434756197 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd71800f040 con 0x7fd7200724e0 2024-03-23T14:35:21.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7201420d0 con 0x7fd7200724e0 2024-03-23T14:35:21.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd71d7fa640 1 -- 172.21.15.161:0/3434756197 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7180146c0 con 0x7fd7200724e0 2024-03-23T14:35:21.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.735+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd720142690 con 0x7fd7200724e0 2024-03-23T14:35:21.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.736+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7201428b0 con 0x7fd7200724e0 2024-03-23T14:35:21.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.736+0000 7fd71d7fa640 1 -- 172.21.15.161:0/3434756197 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd718014820 con 0x7fd7200724e0 2024-03-23T14:35:21.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.737+0000 7fd71d7fa640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd70c06b810 0x7fd70c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:21.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.737+0000 7fd71d7fa640 1 -- 172.21.15.161:0/3434756197 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd71808ac00 con 0x7fd7200724e0 2024-03-23T14:35:21.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.737+0000 7fd724e5f640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd70c06b810 0x7fd70c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:21.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.737+0000 7fd724e5f640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd70c06b810 0x7fd70c06dcd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd710036000 tx=0x7fd710008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:21.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.741+0000 7fd71d7fa640 1 -- 172.21.15.161:0/3434756197 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd718091080 con 0x7fd7200724e0 2024-03-23T14:35:21.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.918+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd7200735e0 con 0x7fd7200724e0 2024-03-23T14:35:21.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.919+0000 7fd71d7fa640 1 -- 172.21.15.161:0/3434756197 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7fd718055650 con 0x7fd7200724e0 2024-03-23T14:35:21.915 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:21.916 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.921+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd70c06b810 msgr2=0x7fd70c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd70c06b810 0x7fd70c06dcd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd710036000 tx=0x7fd710008030 comp rx=0 tx=0).stop 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7200724e0 msgr2=0x7fd7201430f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7200724e0 0x7fd7201430f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd71800efc0 tx=0x7fd71800c490 comp rx=0 tx=0).stop 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 shutdown_connections 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd70c06b810 0x7fd70c06dcd0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7200724e0 0x7fd7201430f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd720072100 0x7fd720142bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 --2- 172.21.15.161:0/3434756197 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd72006eca0 0x7fd7201560f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 >> 172.21.15.161:0/3434756197 conn(0x7fd7200b17f0 msgr2=0x7fd7200b0c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 shutdown_connections 2024-03-23T14:35:21.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:21.922+0000 7fd7270c3640 1 -- 172.21.15.161:0/3434756197 wait complete. 2024-03-23T14:35:21.928 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:21.929 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:22.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.036+0000 7f7547be2640 1 Processor -- start 2024-03-23T14:35:22.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.036+0000 7f7547be2640 1 -- start start 2024-03-23T14:35:22.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f7547be2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f754006c3a0 0x7f75400cc300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f7547be2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75400ae940 0x7f75400aed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f7547be2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f754006c7e0 con 0x7f75400b0c70 2024-03-23T14:35:22.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f7547be2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f75400af2b0 con 0x7f75400ae940 2024-03-23T14:35:22.033 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f7547be2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f75400b1050 con 0x7f754006c3a0 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754617f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75400ae940 0x7f75400aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754617f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75400ae940 0x7f75400aed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33324/0 (socket says 172.21.15.161:33324) 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754597e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f75400b0c70 0x7f75400cda30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57192/0 (socket says 172.21.15.161:57192) 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754617f640 1 -- 172.21.15.161:0/1724058837 learned_addr learned my addr 172.21.15.161:0/1724058837 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754517d640 1 --2- 172.21.15.161:0/1724058837 >> v2:172.21.15.183:3300/0 conn(0x7f754006c3a0 0x7f75400cc300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754617f640 1 -- 172.21.15.161:0/1724058837 >> v2:172.21.15.183:3300/0 conn(0x7f754006c3a0 msgr2=0x7f75400cc300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754617f640 1 --2- 172.21.15.161:0/1724058837 >> v2:172.21.15.183:3300/0 conn(0x7f754006c3a0 0x7f75400cc300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754617f640 1 -- 172.21.15.161:0/1724058837 >> v1:172.21.15.161:6789/0 conn(0x7f75400b0c70 legacy=0x7f75400cda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754617f640 1 -- 172.21.15.161:0/1724058837 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75400cc990 con 0x7f75400ae940 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754497c640 1 -- 172.21.15.161:0/1724058837 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1463845093 0 0) 0x7f753c001110 con 0x7f75400b0c70 2024-03-23T14:35:22.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.037+0000 7f754517d640 1 --2- 172.21.15.161:0/1724058837 >> v2:172.21.15.183:3300/0 conn(0x7f754006c3a0 0x7f75400cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.038+0000 7f754617f640 1 --2- 172.21.15.161:0/1724058837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75400ae940 0x7f75400aed70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f75400d2b20 tx=0x7f753802c010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4766b85db40a735d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.038+0000 7f754497c640 1 -- 172.21.15.161:0/1724058837 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753802cbb0 con 0x7f75400ae940 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.038+0000 7f754497c640 1 -- 172.21.15.161:0/1724058837 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f753802cd10 con 0x7f75400ae940 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.038+0000 7f754497c640 1 -- 172.21.15.161:0/1724058837 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7538034610 con 0x7f75400ae940 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.039+0000 7f7547be2640 1 -- 172.21.15.161:0/1724058837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75400ae940 msgr2=0x7f75400aed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.039+0000 7f7547be2640 1 --2- 172.21.15.161:0/1724058837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75400ae940 0x7f75400aed70 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f75400d2b20 tx=0x7f753802c010 comp rx=0 tx=0).stop 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.039+0000 7f7547be2640 1 -- 172.21.15.161:0/1724058837 shutdown_connections 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.039+0000 7f7547be2640 1 --2- 172.21.15.161:0/1724058837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75400ae940 0x7f75400aed70 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.039+0000 7f7547be2640 1 --2- 172.21.15.161:0/1724058837 >> v2:172.21.15.183:3300/0 conn(0x7f754006c3a0 0x7f75400cc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.039+0000 7f7547be2640 1 -- 172.21.15.161:0/1724058837 >> 172.21.15.161:0/1724058837 conn(0x7f754009aa40 msgr2=0x7f754009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.039+0000 7f7547be2640 1 -- 172.21.15.161:0/1724058837 shutdown_connections 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.039+0000 7f7547be2640 1 -- 172.21.15.161:0/1724058837 wait complete. 2024-03-23T14:35:22.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.040+0000 7f7547be2640 1 Processor -- start 2024-03-23T14:35:22.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.040+0000 7f7547be2640 1 -- start start 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f7547be2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f754006c3a0 0x7f75401f7410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f7547be2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75400ae940 0x7f75400caa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f7547be2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75400b0c70 0x7f754015de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f7547be2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75401f83b0 con 0x7f754006c3a0 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f7547be2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f75401f3df0 con 0x7f75400ae940 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f7547be2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f75401f40c0 con 0x7f75400b0c70 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f754006c3a0 0x7f75401f7410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f754006c3a0 0x7f75401f7410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33340/0 (socket says 172.21.15.161:33340) 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 -- 172.21.15.161:0/3753293131 learned_addr learned my addr 172.21.15.161:0/3753293131 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:22.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754617f640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75400b0c70 0x7f754015de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754517d640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75400ae940 0x7f75400caa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 -- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75400b0c70 msgr2=0x7f754015de50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75400b0c70 0x7f754015de50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 -- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75400ae940 msgr2=0x7f75400caa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75400ae940 0x7f75400caa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 -- 172.21.15.161:0/3753293131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7530009400 con 0x7f754006c3a0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f754597e640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f754006c3a0 0x7f75401f7410 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f753c00c430 tx=0x7f753c00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f75367fc640 1 -- 172.21.15.161:0/3753293131 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c0074e0 con 0x7f754006c3a0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f75367fc640 1 -- 172.21.15.161:0/3753293131 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f753c007660 con 0x7f754006c3a0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7538009b30 con 0x7f754006c3a0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.041+0000 7f75367fc640 1 -- 172.21.15.161:0/3753293131 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f753c004a20 con 0x7f754006c3a0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.042+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75401f7b10 con 0x7f754006c3a0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.042+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7540068680 con 0x7f754006c3a0 2024-03-23T14:35:22.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.043+0000 7f75367fc640 1 -- 172.21.15.161:0/3753293131 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f753c004b80 con 0x7f754006c3a0 2024-03-23T14:35:22.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.043+0000 7f75367fc640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f752406b810 0x7f752406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.043+0000 7f75367fc640 1 -- 172.21.15.161:0/3753293131 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f753c089cb0 con 0x7f754006c3a0 2024-03-23T14:35:22.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.043+0000 7f754517d640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f752406b810 0x7f752406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.044+0000 7f754517d640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f752406b810 0x7f752406dcd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f75400ca710 tx=0x7f7530009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.047+0000 7f75367fc640 1 -- 172.21.15.161:0/3753293131 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f753c054780 con 0x7f754006c3a0 2024-03-23T14:35:22.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.221+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f75400d2bd0 con 0x7f754006c3a0 2024-03-23T14:35:22.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.221+0000 7f75367fc640 1 -- 172.21.15.161:0/3753293131 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f753c057e40 con 0x7f754006c3a0 2024-03-23T14:35:22.217 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f752406b810 msgr2=0x7f752406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f752406b810 0x7f752406dcd0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f75400ca710 tx=0x7f7530009190 comp rx=0 tx=0).stop 2024-03-23T14:35:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f754006c3a0 msgr2=0x7f75401f7410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f754006c3a0 0x7f75401f7410 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f753c00c430 tx=0x7f753c00c8f0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 shutdown_connections 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f752406b810 0x7f752406dcd0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75400b0c70 0x7f754015de50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75400ae940 0x7f75400caa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 --2- 172.21.15.161:0/3753293131 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f754006c3a0 0x7f75401f7410 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 >> 172.21.15.161:0/3753293131 conn(0x7f754009aa40 msgr2=0x7f75400b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 shutdown_connections 2024-03-23T14:35:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.223+0000 7f7547be2640 1 -- 172.21.15.161:0/3753293131 wait complete. 2024-03-23T14:35:22.230 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:22.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.337+0000 7ff05742a640 1 Processor -- start 2024-03-23T14:35:22.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.337+0000 7ff05742a640 1 -- start start 2024-03-23T14:35:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff05742a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015de70 0x7ff05015e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff05742a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff0500d2130 0x7ff05006c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff05742a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0500b3710 con 0x7ff05015e790 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff05742a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7ff05015ecd0 con 0x7ff0500d2130 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff05742a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff05015efb0 con 0x7ff05015de70 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff0551c6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015de70 0x7ff05015e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff0559c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff0500d2130 0x7ff05006c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff0551c6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015de70 0x7ff05015e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:55904/0 (socket says 172.21.15.161:55904) 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff0559c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff0500d2130 0x7ff05006c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:44892/0 (socket says 172.21.15.161:44892) 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff0551c6640 1 -- 172.21.15.161:0/2645961966 learned_addr learned my addr 172.21.15.161:0/2645961966 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.338+0000 7ff0549c5640 1 --1- 172.21.15.161:0/2645961966 >> v1:172.21.15.183:6789/0 conn(0x7ff05015e790 0x7ff0500d1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:58694/0 (socket says 172.21.15.161:58694) 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff0551c6640 1 -- 172.21.15.161:0/2645961966 >> v2:172.21.15.183:3300/0 conn(0x7ff0500d2130 msgr2=0x7ff05006c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff0551c6640 1 --2- 172.21.15.161:0/2645961966 >> v2:172.21.15.183:3300/0 conn(0x7ff0500d2130 0x7ff05006c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff0551c6640 1 -- 172.21.15.161:0/2645961966 >> v1:172.21.15.183:6789/0 conn(0x7ff05015e790 legacy=0x7ff0500d1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff0551c6640 1 -- 172.21.15.161:0/2645961966 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05015f290 con 0x7ff05015de70 2024-03-23T14:35:22.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff0559c7640 1 --2- 172.21.15.161:0/2645961966 >> v2:172.21.15.183:3300/0 conn(0x7ff0500d2130 0x7ff05006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff0551c6640 1 --2- 172.21.15.161:0/2645961966 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015de70 0x7ff05015e250 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff04c009e50 tx=0x7ff04c02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58a90853a020c5d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff047fff640 1 -- 172.21.15.161:0/2645961966 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c02cb00 con 0x7ff05015de70 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff047fff640 1 -- 172.21.15.161:0/2645961966 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff04c02cc60 con 0x7ff05015de70 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.339+0000 7ff047fff640 1 -- 172.21.15.161:0/2645961966 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff04c0343f0 con 0x7ff05015de70 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.340+0000 7ff05742a640 1 -- 172.21.15.161:0/2645961966 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015de70 msgr2=0x7ff05015e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.340+0000 7ff05742a640 1 --2- 172.21.15.161:0/2645961966 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015de70 0x7ff05015e250 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff04c009e50 tx=0x7ff04c02bf60 comp rx=0 tx=0).stop 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.340+0000 7ff05742a640 1 -- 172.21.15.161:0/2645961966 shutdown_connections 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.340+0000 7ff05742a640 1 --2- 172.21.15.161:0/2645961966 >> v2:172.21.15.183:3300/0 conn(0x7ff0500d2130 0x7ff05006c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.340+0000 7ff05742a640 1 --2- 172.21.15.161:0/2645961966 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015de70 0x7ff05015e250 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.340+0000 7ff05742a640 1 -- 172.21.15.161:0/2645961966 >> 172.21.15.161:0/2645961966 conn(0x7ff05009aa40 msgr2=0x7ff05009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:22.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.340+0000 7ff05742a640 1 -- 172.21.15.161:0/2645961966 shutdown_connections 2024-03-23T14:35:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.340+0000 7ff05742a640 1 -- 172.21.15.161:0/2645961966 wait complete. 2024-03-23T14:35:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.341+0000 7ff05742a640 1 Processor -- start 2024-03-23T14:35:22.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.341+0000 7ff05742a640 1 -- start start 2024-03-23T14:35:22.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff05742a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0500d2130 0x7ff0500b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff05742a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05015de70 0x7ff05014edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff05742a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015e790 0x7ff05014f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff05742a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff0500b1f90 con 0x7ff05015de70 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff05742a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff0500b21e0 con 0x7ff0500d2130 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff05742a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff0500af6c0 con 0x7ff05015e790 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05015de70 0x7ff05014edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05015de70 0x7ff05014edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33352/0 (socket says 172.21.15.161:33352) 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0551c6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0500d2130 0x7ff0500b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 -- 172.21.15.161:0/3892984701 learned_addr learned my addr 172.21.15.161:0/3892984701 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0559c7640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015e790 0x7ff05014f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 -- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015e790 msgr2=0x7ff05014f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015e790 0x7ff05014f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 -- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0500d2130 msgr2=0x7ff0500b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0500d2130 0x7ff0500b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 -- 172.21.15.161:0/3892984701 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff048009400 con 0x7ff05015de70 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0551c6640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0500d2130 0x7ff0500b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.342+0000 7ff0549c5640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05015de70 0x7ff05014edc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff04000c3c0 tx=0x7ff04000c7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.343+0000 7ff045ffb640 1 -- 172.21.15.161:0/3892984701 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0400074e0 con 0x7ff05015de70 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.343+0000 7ff045ffb640 1 -- 172.21.15.161:0/3892984701 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff040007640 con 0x7ff05015de70 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.343+0000 7ff045ffb640 1 -- 172.21.15.161:0/3892984701 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff040004a00 con 0x7ff05015de70 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.343+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff04c009b30 con 0x7ff05015de70 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.343+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0500afc80 con 0x7ff05015de70 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.344+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff05006bbb0 con 0x7ff05015de70 2024-03-23T14:35:22.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.344+0000 7ff045ffb640 1 -- 172.21.15.161:0/3892984701 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff040004b60 con 0x7ff05015de70 2024-03-23T14:35:22.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.344+0000 7ff045ffb640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff03406b810 0x7ff03406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.345+0000 7ff0551c6640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff03406b810 0x7ff03406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.345+0000 7ff045ffb640 1 -- 172.21.15.161:0/3892984701 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff040089a50 con 0x7ff05015de70 2024-03-23T14:35:22.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.345+0000 7ff0551c6640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff03406b810 0x7ff03406dcd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff04c035000 tx=0x7ff04c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.349+0000 7ff045ffb640 1 -- 172.21.15.161:0/3892984701 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff040054470 con 0x7ff05015de70 2024-03-23T14:35:22.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.522+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff05004f550 con 0x7ff05015de70 2024-03-23T14:35:22.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.523+0000 7ff045ffb640 1 -- 172.21.15.161:0/3892984701 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7ff040057b30 con 0x7ff05015de70 2024-03-23T14:35:22.519 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:22.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff03406b810 msgr2=0x7ff03406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff03406b810 0x7ff03406dcd0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff04c035000 tx=0x7ff04c030000 comp rx=0 tx=0).stop 2024-03-23T14:35:22.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05015de70 msgr2=0x7ff05014edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05015de70 0x7ff05014edc0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff04000c3c0 tx=0x7ff04000c7c0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 shutdown_connections 2024-03-23T14:35:22.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff03406b810 0x7ff03406dcd0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05015e790 0x7ff05014f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05015de70 0x7ff05014edc0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 --2- 172.21.15.161:0/3892984701 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff0500d2130 0x7ff0500b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 >> 172.21.15.161:0/3892984701 conn(0x7ff05009aa40 msgr2=0x7ff0500cd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:22.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 shutdown_connections 2024-03-23T14:35:22.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.525+0000 7ff05742a640 1 -- 172.21.15.161:0/3892984701 wait complete. 2024-03-23T14:35:22.531 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:22.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.639+0000 7efc9f1b6640 1 Processor -- start 2024-03-23T14:35:22.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.639+0000 7efc9f1b6640 1 -- start start 2024-03-23T14:35:22.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc9f1b6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 0x7efc980ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc9f1b6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efc980cc540 0x7efc980cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc9f1b6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efc980ca8e0 0x7efc980cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc9f1b6640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efc980cf4b0 con 0x7efc980cc540 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc9f1b6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7efc980cfd10 con 0x7efc980ca8e0 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc9f1b6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efc9802c670 con 0x7efc980b1720 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc97fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efc980cc540 0x7efc980cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc97fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efc980cc540 0x7efc980cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33356/0 (socket says 172.21.15.161:33356) 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc97fff640 1 -- 172.21.15.161:0/114844961 learned_addr learned my addr 172.21.15.161:0/114844961 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc9cf52640 1 --2- 172.21.15.161:0/114844961 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 0x7efc980ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc9d753640 1 --2- 172.21.15.161:0/114844961 >> v2:172.21.15.183:3300/0 conn(0x7efc980ca8e0 0x7efc980cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc97fff640 1 -- 172.21.15.161:0/114844961 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 msgr2=0x7efc980ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc97fff640 1 --2- 172.21.15.161:0/114844961 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 0x7efc980ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc97fff640 1 -- 172.21.15.161:0/114844961 >> v2:172.21.15.183:3300/0 conn(0x7efc980ca8e0 msgr2=0x7efc980cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc97fff640 1 --2- 172.21.15.161:0/114844961 >> v2:172.21.15.183:3300/0 conn(0x7efc980ca8e0 0x7efc980cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.640+0000 7efc97fff640 1 -- 172.21.15.161:0/114844961 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc980715a0 con 0x7efc980cc540 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.641+0000 7efc97fff640 1 --2- 172.21.15.161:0/114844961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980cd1e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc98071290 tx=0x7efc8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8301384b7e6321a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.641+0000 7efc977fe640 1 -- 172.21.15.161:0/114844961 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c031870 con 0x7efc980cc540 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.641+0000 7efc977fe640 1 -- 172.21.15.161:0/114844961 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc8c0319d0 con 0x7efc980cc540 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.641+0000 7efc977fe640 1 -- 172.21.15.161:0/114844961 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8c031d60 con 0x7efc980cc540 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 -- 172.21.15.161:0/114844961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 msgr2=0x7efc980cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/114844961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980cd1e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7efc98071290 tx=0x7efc8c02b910 comp rx=0 tx=0).stop 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 -- 172.21.15.161:0/114844961 shutdown_connections 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/114844961 >> v2:172.21.15.183:3300/0 conn(0x7efc980ca8e0 0x7efc980cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/114844961 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980cd1e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/114844961 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 0x7efc980ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 -- 172.21.15.161:0/114844961 >> 172.21.15.161:0/114844961 conn(0x7efc980b1e60 msgr2=0x7efc980b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:22.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 -- 172.21.15.161:0/114844961 shutdown_connections 2024-03-23T14:35:22.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.642+0000 7efc9f1b6640 1 -- 172.21.15.161:0/114844961 wait complete. 2024-03-23T14:35:22.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.643+0000 7efc9f1b6640 1 Processor -- start 2024-03-23T14:35:22.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.643+0000 7efc9f1b6640 1 -- start start 2024-03-23T14:35:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9f1b6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 0x7efc981e8980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9f1b6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc980ca8e0 0x7efc98161480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9f1b6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9f1b6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efc981e8ec0 con 0x7efc980cc540 2024-03-23T14:35:22.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9f1b6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efc981e78e0 con 0x7efc980ca8e0 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9f1b6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efc981e7bc0 con 0x7efc980b1720 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980b5400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33368/0 (socket says 172.21.15.161:33368) 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 -- 172.21.15.161:0/2010070115 learned_addr learned my addr 172.21.15.161:0/2010070115 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9cf52640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 0x7efc981e8980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc97fff640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc980ca8e0 0x7efc98161480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 -- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 msgr2=0x7efc981e8980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 0x7efc981e8980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 -- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc980ca8e0 msgr2=0x7efc98161480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc980ca8e0 0x7efc98161480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 -- 172.21.15.161:0/2010070115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8c0094d0 con 0x7efc980cc540 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.644+0000 7efc9d753640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980b5400 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efc9000efc0 tx=0x7efc9000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.645+0000 7efc957fa640 1 -- 172.21.15.161:0/2010070115 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90014480 con 0x7efc980cc540 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.645+0000 7efc957fa640 1 -- 172.21.15.161:0/2010070115 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc9000f040 con 0x7efc980cc540 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.645+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc981e7ea0 con 0x7efc980cc540 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.645+0000 7efc957fa640 1 -- 172.21.15.161:0/2010070115 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc90014820 con 0x7efc980cc540 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.645+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc981e8460 con 0x7efc980cc540 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.646+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc981e8680 con 0x7efc980cc540 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.646+0000 7efc957fa640 1 -- 172.21.15.161:0/2010070115 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efc90014980 con 0x7efc980cc540 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.646+0000 7efc957fa640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efc8406b810 0x7efc8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.647+0000 7efc957fa640 1 -- 172.21.15.161:0/2010070115 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efc9008ac20 con 0x7efc980cc540 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.647+0000 7efc9cf52640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efc8406b810 0x7efc8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.647+0000 7efc9cf52640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efc8406b810 0x7efc8406dcd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efc981ea050 tx=0x7efc88009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.651+0000 7efc957fa640 1 -- 172.21.15.161:0/2010070115 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efc900556f0 con 0x7efc980cc540 2024-03-23T14:35:22.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.824+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efc9804f4e0 con 0x7efc980cc540 2024-03-23T14:35:22.821 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.825+0000 7efc957fa640 1 -- 172.21.15.161:0/2010070115 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7efc900180a0 con 0x7efc980cc540 2024-03-23T14:35:22.821 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efc8406b810 msgr2=0x7efc8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efc8406b810 0x7efc8406dcd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7efc981ea050 tx=0x7efc88009190 comp rx=0 tx=0).stop 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 msgr2=0x7efc980b5400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980b5400 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efc9000efc0 tx=0x7efc9000c490 comp rx=0 tx=0).stop 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 shutdown_connections 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efc8406b810 0x7efc8406dcd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efc980cc540 0x7efc980b5400 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efc980ca8e0 0x7efc98161480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 --2- 172.21.15.161:0/2010070115 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efc980b1720 0x7efc981e8980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 >> 172.21.15.161:0/2010070115 conn(0x7efc980b1e60 msgr2=0x7efc9815eb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 shutdown_connections 2024-03-23T14:35:22.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.827+0000 7efc9f1b6640 1 -- 172.21.15.161:0/2010070115 wait complete. 2024-03-23T14:35:22.833 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:22.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.939+0000 7fc32fdc7640 1 Processor -- start 2024-03-23T14:35:22.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.939+0000 7fc32fdc7640 1 -- start start 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32fdc7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc328072100 0x7fc32815b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32fdc7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc3280724e0 0x7fc32806e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32fdc7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc32815be70 con 0x7fc32815b930 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32fdc7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc32815c430 con 0x7fc328072100 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32fdc7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc32815c150 con 0x7fc3280724e0 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32db63640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc328072100 0x7fc32815b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32db63640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc328072100 0x7fc32815b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33380/0 (socket says 172.21.15.161:33380) 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32e364640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc32815b930 0x7fc32815e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57196/0 (socket says 172.21.15.161:57196) 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32db63640 1 -- 172.21.15.161:0/2310004069 learned_addr learned my addr 172.21.15.161:0/2310004069 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:22.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32d362640 1 --2- 172.21.15.161:0/2310004069 >> v2:172.21.15.183:3300/0 conn(0x7fc3280724e0 0x7fc32806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.940+0000 7fc32db63640 1 -- 172.21.15.161:0/2310004069 >> v2:172.21.15.183:3300/0 conn(0x7fc3280724e0 msgr2=0x7fc32806e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:22.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.941+0000 7fc32db63640 1 --2- 172.21.15.161:0/2310004069 >> v2:172.21.15.183:3300/0 conn(0x7fc3280724e0 0x7fc32806e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.941+0000 7fc32db63640 1 -- 172.21.15.161:0/2310004069 >> v1:172.21.15.161:6789/0 conn(0x7fc32815b930 legacy=0x7fc32815e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.941+0000 7fc32db63640 1 -- 172.21.15.161:0/2310004069 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3280b2c70 con 0x7fc328072100 2024-03-23T14:35:22.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.941+0000 7fc32db63640 1 --2- 172.21.15.161:0/2310004069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc32815b3f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc328073420 tx=0x7fc32402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d381ca918e306c00 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.941+0000 7fc32cb61640 1 -- 172.21.15.161:0/2310004069 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324030860 con 0x7fc328072100 2024-03-23T14:35:22.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.941+0000 7fc32cb61640 1 -- 172.21.15.161:0/2310004069 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3240309c0 con 0x7fc328072100 2024-03-23T14:35:22.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.941+0000 7fc32cb61640 1 -- 172.21.15.161:0/2310004069 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324030d50 con 0x7fc328072100 2024-03-23T14:35:22.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.942+0000 7fc32fdc7640 1 -- 172.21.15.161:0/2310004069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 msgr2=0x7fc32815b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.942+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/2310004069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc32815b3f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc328073420 tx=0x7fc32402b9a0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.942+0000 7fc32fdc7640 1 -- 172.21.15.161:0/2310004069 shutdown_connections 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.942+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/2310004069 >> v2:172.21.15.183:3300/0 conn(0x7fc3280724e0 0x7fc32806e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.942+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/2310004069 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc32815b3f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.942+0000 7fc32fdc7640 1 -- 172.21.15.161:0/2310004069 >> 172.21.15.161:0/2310004069 conn(0x7fc3280b17f0 msgr2=0x7fc3280b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.942+0000 7fc32fdc7640 1 -- 172.21.15.161:0/2310004069 shutdown_connections 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.942+0000 7fc32fdc7640 1 -- 172.21.15.161:0/2310004069 wait complete. 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.943+0000 7fc32fdc7640 1 Processor -- start 2024-03-23T14:35:22.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.943+0000 7fc32fdc7640 1 -- start start 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32fdc7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc328145430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32fdc7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280724e0 0x7fc3280b33a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32fdc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc32815b930 0x7fc3280b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32fdc7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc32815a920 con 0x7fc328072100 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32fdc7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc328159880 con 0x7fc3280724e0 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32fdc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc328159b60 con 0x7fc32815b930 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc328145430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc328145430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33390/0 (socket says 172.21.15.161:33390) 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 -- 172.21.15.161:0/1283633940 learned_addr learned my addr 172.21.15.161:0/1283633940 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:22.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32d362640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280724e0 0x7fc3280b33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32e364640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc32815b930 0x7fc3280b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 -- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc32815b930 msgr2=0x7fc3280b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc32815b930 0x7fc3280b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 -- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280724e0 msgr2=0x7fc3280b33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280724e0 0x7fc3280b33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 -- 172.21.15.161:0/1283633940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3240094d0 con 0x7fc328072100 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.944+0000 7fc32db63640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc328145430 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc32402b9e0 tx=0x7fc324004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.945+0000 7fc31e7fc640 1 -- 172.21.15.161:0/1283633940 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324039c90 con 0x7fc328072100 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.945+0000 7fc31e7fc640 1 -- 172.21.15.161:0/1283633940 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc324039e10 con 0x7fc328072100 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.945+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc328159e40 con 0x7fc328072100 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.945+0000 7fc31e7fc640 1 -- 172.21.15.161:0/1283633940 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc324043640 con 0x7fc328072100 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.945+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32815a400 con 0x7fc328072100 2024-03-23T14:35:22.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.945+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3280735e0 con 0x7fc328072100 2024-03-23T14:35:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.946+0000 7fc31e7fc640 1 -- 172.21.15.161:0/1283633940 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc3240437a0 con 0x7fc328072100 2024-03-23T14:35:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.946+0000 7fc31e7fc640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc30c06b810 0x7fc30c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.946+0000 7fc31e7fc640 1 -- 172.21.15.161:0/1283633940 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc32403e070 con 0x7fc328072100 2024-03-23T14:35:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.947+0000 7fc32d362640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc30c06b810 0x7fc30c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:22.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.947+0000 7fc32d362640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc30c06b810 0x7fc30c06dcd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc328154a50 tx=0x7fc318009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:22.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:22.951+0000 7fc31e7fc640 1 -- 172.21.15.161:0/1283633940 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc324077c90 con 0x7fc328072100 2024-03-23T14:35:23.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.160+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc32804f550 con 0x7fc328072100 2024-03-23T14:35:23.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.160+0000 7fc31e7fc640 1 -- 172.21.15.161:0/1283633940 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5181 (secure 0 0 0) 0x7fc32407b350 con 0x7fc328072100 2024-03-23T14:35:23.157 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:23.157 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:35:13.112471+0000","modified":"2024-03-23T14:35:15.108804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-23T14:35:23.157 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 6 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc30c06b810 msgr2=0x7fc30c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc30c06b810 0x7fc30c06dcd0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fc328154a50 tx=0x7fc318009190 comp rx=0 tx=0).stop 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 msgr2=0x7fc328145430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc328145430 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fc32402b9e0 tx=0x7fc324004730 comp rx=0 tx=0).stop 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 shutdown_connections 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc30c06b810 0x7fc30c06dcd0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc32815b930 0x7fc3280b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280724e0 0x7fc3280b33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 --2- 172.21.15.161:0/1283633940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc328072100 0x7fc328145430 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 >> 172.21.15.161:0/1283633940 conn(0x7fc3280b17f0 msgr2=0x7fc3280cf260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:23.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 shutdown_connections 2024-03-23T14:35:23.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.162+0000 7fc32fdc7640 1 -- 172.21.15.161:0/1283633940 wait complete. 2024-03-23T14:35:23.169 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:23.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.274+0000 7f938d609640 1 Processor -- start 2024-03-23T14:35:23.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.274+0000 7f938d609640 1 -- start start 2024-03-23T14:35:23.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f938d609640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cec90 0x7f93881690f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f938d609640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93880701e0 con 0x7f93880cdf10 2024-03-23T14:35:23.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f938d609640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93880d0af0 con 0x7f93880d0f80 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f938d609640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f93880cfee0 con 0x7f93880cec90 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f9386ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f93880cdf10 0x7f938806f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57202/0 (socket says 172.21.15.161:57202) 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f9386ffd640 1 -- 172.21.15.161:0/2594588937 learned_addr learned my addr 172.21.15.161:0/2594588937 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f93877fe640 1 --2- 172.21.15.161:0/2594588937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cec90 0x7f93881690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f9385ffb640 1 -- 172.21.15.161:0/2594588937 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1108137726 0 0) 0x7f93880701e0 con 0x7f93880cdf10 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f9385ffb640 1 -- 172.21.15.161:0/2594588937 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9370003520 con 0x7f93880cdf10 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f9385ffb640 1 -- 172.21.15.161:0/2594588937 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1164467554 0 0) 0x7f93880d0af0 con 0x7f93880d0f80 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f9385ffb640 1 -- 172.21.15.161:0/2594588937 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f93880701e0 con 0x7f93880d0f80 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f93877fe640 1 -- 172.21.15.161:0/2594588937 >> v1:172.21.15.183:6789/0 conn(0x7f93880d0f80 legacy=0x7f93880ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f93877fe640 1 -- 172.21.15.161:0/2594588937 >> v1:172.21.15.161:6789/0 conn(0x7f93880cdf10 legacy=0x7f938806f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.275+0000 7f93877fe640 1 -- 172.21.15.161:0/2594588937 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93880d01c0 con 0x7f93880cec90 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.276+0000 7f93877fe640 1 --2- 172.21.15.161:0/2594588937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cec90 0x7f93881690f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9380009dc0 tx=0x7f938002bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=145c7057b035d7fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.276+0000 7f9385ffb640 1 -- 172.21.15.161:0/2594588937 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938002ccd0 con 0x7f93880cec90 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.276+0000 7f9385ffb640 1 -- 172.21.15.161:0/2594588937 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f938002ce30 con 0x7f93880cec90 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.276+0000 7f9385ffb640 1 -- 172.21.15.161:0/2594588937 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9380034660 con 0x7f93880cec90 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.277+0000 7f938d609640 1 -- 172.21.15.161:0/2594588937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cec90 msgr2=0x7f93881690f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.277+0000 7f938d609640 1 --2- 172.21.15.161:0/2594588937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cec90 0x7f93881690f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f9380009dc0 tx=0x7f938002bf20 comp rx=0 tx=0).stop 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.277+0000 7f938d609640 1 -- 172.21.15.161:0/2594588937 shutdown_connections 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.277+0000 7f938d609640 1 --2- 172.21.15.161:0/2594588937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cec90 0x7f93881690f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.277+0000 7f938d609640 1 -- 172.21.15.161:0/2594588937 >> 172.21.15.161:0/2594588937 conn(0x7f93880d1d00 msgr2=0x7f93880d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:23.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.277+0000 7f938d609640 1 -- 172.21.15.161:0/2594588937 shutdown_connections 2024-03-23T14:35:23.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.277+0000 7f938d609640 1 -- 172.21.15.161:0/2594588937 wait complete. 2024-03-23T14:35:23.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.278+0000 7f938d609640 1 Processor -- start 2024-03-23T14:35:23.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.278+0000 7f938d609640 1 -- start start 2024-03-23T14:35:23.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f938d609640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cdf10 0x7f9388159440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f938d609640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93880cec90 0x7f93881562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f938d609640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93880d0f80 0x7f938815cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f938d609640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f938815a090 con 0x7f93880d0f80 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f938d609640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f938815d300 con 0x7f93880cec90 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f938d609640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9388159ad0 con 0x7f93880cdf10 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93877fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93880d0f80 0x7f938815cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93877fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93880d0f80 0x7f938815cdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33406/0 (socket says 172.21.15.161:33406) 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93877fe640 1 -- 172.21.15.161:0/369199341 learned_addr learned my addr 172.21.15.161:0/369199341 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f9386ffd640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cdf10 0x7f9388159440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93867fc640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93880cec90 0x7f93881562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93877fe640 1 -- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cdf10 msgr2=0x7f9388159440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93877fe640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cdf10 0x7f9388159440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93877fe640 1 -- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93880cec90 msgr2=0x7f93881562e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93877fe640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93880cec90 0x7f93881562e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.279+0000 7f93877fe640 1 -- 172.21.15.161:0/369199341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9380009a70 con 0x7f93880d0f80 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.280+0000 7f93877fe640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93880d0f80 0x7f938815cdc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9380037040 tx=0x7f9380004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.280+0000 7f936f7fe640 1 -- 172.21.15.161:0/369199341 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938003d600 con 0x7f93880d0f80 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.280+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9388159db0 con 0x7f93880d0f80 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.280+0000 7f936f7fe640 1 -- 172.21.15.161:0/369199341 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f938003d760 con 0x7f93880d0f80 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.280+0000 7f936f7fe640 1 -- 172.21.15.161:0/369199341 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f938003db10 con 0x7f93880d0f80 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.280+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93881571d0 con 0x7f93880d0f80 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.281+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9388156d50 con 0x7f93880d0f80 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.281+0000 7f936f7fe640 1 -- 172.21.15.161:0/369199341 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f938003b040 con 0x7f93880d0f80 2024-03-23T14:35:23.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.282+0000 7f936f7fe640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f937406b810 0x7f937406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.282+0000 7f936f7fe640 1 -- 172.21.15.161:0/369199341 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f93800ac330 con 0x7f93880d0f80 2024-03-23T14:35:23.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.282+0000 7f9386ffd640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f937406b810 0x7f937406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.282+0000 7f9386ffd640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f937406b810 0x7f937406dcd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9378000f60 tx=0x7f9378007d70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:23.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.286+0000 7f936f7fe640 1 -- 172.21.15.161:0/369199341 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9388156d50 con 0x7f93880d0f80 2024-03-23T14:35:23.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.459+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9388156d50 con 0x7f93880d0f80 2024-03-23T14:35:23.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.460+0000 7f936f7fe640 1 -- 172.21.15.161:0/369199341 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f9388156d50 con 0x7f93880d0f80 2024-03-23T14:35:23.456 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:23.456 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:23.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f937406b810 msgr2=0x7f937406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f937406b810 0x7f937406dcd0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9378000f60 tx=0x7f9378007d70 comp rx=0 tx=0).stop 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93880d0f80 msgr2=0x7f938815cdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93880d0f80 0x7f938815cdc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9380037040 tx=0x7f9380004a40 comp rx=0 tx=0).stop 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 shutdown_connections 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f937406b810 0x7f937406dcd0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f93880d0f80 0x7f938815cdc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f93880cec90 0x7f93881562e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 --2- 172.21.15.161:0/369199341 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f93880cdf10 0x7f9388159440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 >> 172.21.15.161:0/369199341 conn(0x7f93880d1d00 msgr2=0x7f93880ae9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 shutdown_connections 2024-03-23T14:35:23.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.462+0000 7f938d609640 1 -- 172.21.15.161:0/369199341 wait complete. 2024-03-23T14:35:23.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:23.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.573+0000 7fce6d815640 1 Processor -- start 2024-03-23T14:35:23.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.573+0000 7fce6d815640 1 -- start start 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce6d815640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fce680ad850 0x7fce680cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce6d815640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fce680cd380 0x7fce68168810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce6d815640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce6802c1c0 con 0x7fce680cf840 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce6d815640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fce6802c650 con 0x7fce680cd380 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce6d815640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fce6815e620 con 0x7fce680ad850 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce677fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fce680cd380 0x7fce68168810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce677fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fce680cd380 0x7fce68168810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33412/0 (socket says 172.21.15.161:33412) 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce677fe640 1 -- 172.21.15.161:0/305329848 learned_addr learned my addr 172.21.15.161:0/305329848 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:23.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce667fc640 1 --2- 172.21.15.161:0/305329848 >> v2:172.21.15.183:3300/0 conn(0x7fce680ad850 0x7fce680cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce677fe640 1 -- 172.21.15.161:0/305329848 >> v2:172.21.15.183:3300/0 conn(0x7fce680ad850 msgr2=0x7fce680cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce677fe640 1 --2- 172.21.15.161:0/305329848 >> v2:172.21.15.183:3300/0 conn(0x7fce680ad850 0x7fce680cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce677fe640 1 -- 172.21.15.161:0/305329848 >> v1:172.21.15.183:6789/0 conn(0x7fce680cf840 legacy=0x7fce680ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.574+0000 7fce677fe640 1 -- 172.21.15.161:0/305329848 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce68071630 con 0x7fce680cd380 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.575+0000 7fce667fc640 1 --2- 172.21.15.161:0/305329848 >> v2:172.21.15.183:3300/0 conn(0x7fce680ad850 0x7fce680cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.575+0000 7fce677fe640 1 --2- 172.21.15.161:0/305329848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cd380 0x7fce68168810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce680d0210 tx=0x7fce6002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e5bdc92a86e10d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.575+0000 7fce65ffb640 1 -- 172.21.15.161:0/305329848 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60030860 con 0x7fce680cd380 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.575+0000 7fce65ffb640 1 -- 172.21.15.161:0/305329848 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce600309c0 con 0x7fce680cd380 2024-03-23T14:35:23.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.575+0000 7fce65ffb640 1 -- 172.21.15.161:0/305329848 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60030d50 con 0x7fce680cd380 2024-03-23T14:35:23.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.576+0000 7fce6d815640 1 -- 172.21.15.161:0/305329848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cd380 msgr2=0x7fce68168810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.576+0000 7fce6d815640 1 --2- 172.21.15.161:0/305329848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cd380 0x7fce68168810 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fce680d0210 tx=0x7fce6002b9a0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.576+0000 7fce6d815640 1 -- 172.21.15.161:0/305329848 shutdown_connections 2024-03-23T14:35:23.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.576+0000 7fce6d815640 1 --2- 172.21.15.161:0/305329848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cd380 0x7fce68168810 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.576+0000 7fce6d815640 1 --2- 172.21.15.161:0/305329848 >> v2:172.21.15.183:3300/0 conn(0x7fce680ad850 0x7fce680cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.576+0000 7fce6d815640 1 -- 172.21.15.161:0/305329848 >> 172.21.15.161:0/305329848 conn(0x7fce680b1e60 msgr2=0x7fce680b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:23.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.576+0000 7fce6d815640 1 -- 172.21.15.161:0/305329848 shutdown_connections 2024-03-23T14:35:23.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.576+0000 7fce6d815640 1 -- 172.21.15.161:0/305329848 wait complete. 2024-03-23T14:35:23.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.577+0000 7fce6d815640 1 Processor -- start 2024-03-23T14:35:23.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.577+0000 7fce6d815640 1 -- start start 2024-03-23T14:35:23.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce6d815640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce680ad850 0x7fce6814f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce6d815640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce680cd380 0x7fce680c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce6d815640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cf840 0x7fce680af6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce6d815640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fce680b00b0 con 0x7fce680cf840 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce6d815640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fce680b0390 con 0x7fce680cd380 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce6d815640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fce680b0670 con 0x7fce680ad850 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce677fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cf840 0x7fce680af6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce677fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cf840 0x7fce680af6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33414/0 (socket says 172.21.15.161:33414) 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce677fe640 1 -- 172.21.15.161:0/1278452884 learned_addr learned my addr 172.21.15.161:0/1278452884 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce66ffd640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce680ad850 0x7fce6814f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce667fc640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce680cd380 0x7fce680c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce677fe640 1 -- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce680ad850 msgr2=0x7fce6814f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce677fe640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce680ad850 0x7fce6814f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce677fe640 1 -- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce680cd380 msgr2=0x7fce680c9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce677fe640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce680cd380 0x7fce680c9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.578+0000 7fce677fe640 1 -- 172.21.15.161:0/1278452884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce5c009400 con 0x7fce680cf840 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.579+0000 7fce677fe640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cf840 0x7fce680af6d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fce6002b9e0 tx=0x7fce60004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.579+0000 7fce4ffff640 1 -- 172.21.15.161:0/1278452884 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60039c90 con 0x7fce680cf840 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.579+0000 7fce4ffff640 1 -- 172.21.15.161:0/1278452884 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce60039e10 con 0x7fce680cf840 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.579+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce600094d0 con 0x7fce680cf840 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.579+0000 7fce4ffff640 1 -- 172.21.15.161:0/1278452884 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce60043640 con 0x7fce680cf840 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.579+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce680b4f80 con 0x7fce680cf840 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.580+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce680b4b00 con 0x7fce680cf840 2024-03-23T14:35:23.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.580+0000 7fce4ffff640 1 -- 172.21.15.161:0/1278452884 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fce600437a0 con 0x7fce680cf840 2024-03-23T14:35:23.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.581+0000 7fce4ffff640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce5406b810 0x7fce5406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:23.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.581+0000 7fce4ffff640 1 -- 172.21.15.161:0/1278452884 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fce6003e070 con 0x7fce680cf840 2024-03-23T14:35:23.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.581+0000 7fce66ffd640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce5406b810 0x7fce5406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:23.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.581+0000 7fce66ffd640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce5406b810 0x7fce5406dcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce58001e70 tx=0x7fce58009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:23.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.585+0000 7fce4ffff640 1 -- 172.21.15.161:0/1278452884 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fce680b4b00 con 0x7fce680cf840 2024-03-23T14:35:23.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.792+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce680b4b00 con 0x7fce680cf840 2024-03-23T14:35:23.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.793+0000 7fce4ffff640 1 -- 172.21.15.161:0/1278452884 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5181 (secure 0 0 0) 0x7fce680b4b00 con 0x7fce680cf840 2024-03-23T14:35:23.788 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:23.789 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:35:13.112471+0000","modified":"2024-03-23T14:35:15.108804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-23T14:35:23.789 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 6 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce5406b810 msgr2=0x7fce5406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce5406b810 0x7fce5406dcd0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce58001e70 tx=0x7fce58009040 comp rx=0 tx=0).stop 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cf840 msgr2=0x7fce680af6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cf840 0x7fce680af6d0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fce6002b9e0 tx=0x7fce60004730 comp rx=0 tx=0).stop 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 shutdown_connections 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce5406b810 0x7fce5406dcd0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fce680cf840 0x7fce680af6d0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fce680cd380 0x7fce680c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 --2- 172.21.15.161:0/1278452884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fce680ad850 0x7fce6814f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.794+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 >> 172.21.15.161:0/1278452884 conn(0x7fce680b1e60 msgr2=0x7fce680ca8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.795+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 shutdown_connections 2024-03-23T14:35:23.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:23.795+0000 7fce6d815640 1 -- 172.21.15.161:0/1278452884 wait complete. 2024-03-23T14:35:24.205 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 6, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T14:35:13.112471+0000', 'modified': '2024-03-23T14:35:15.108804+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4282}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4282': {'gid': 4282, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.183:6837/1031097819', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6835', 'nonce': 1031097819}, {'type': 'v1', 'addr': '172.21.15.183:6837', 'nonce': 1031097819}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 4282, 'qdb_cluster': [4282]} 2024-03-23T14:35:24.205 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T14:35:24.205 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-23T14:35:24.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.312+0000 7fa10a447640 1 Processor -- start 2024-03-23T14:35:24.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.312+0000 7fa10a447640 1 -- start start 2024-03-23T14:35:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa10a447640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa10415e400 0x7fa1040cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa10a447640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1040d0460 con 0x7fa1040d2b40 2024-03-23T14:35:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa10a447640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1040d06b0 con 0x7fa1040cf9d0 2024-03-23T14:35:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa10a447640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa1040d1050 con 0x7fa10415e400 2024-03-23T14:35:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa103fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa10415e400 0x7fa1040cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa1089e4640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa1040d2b40 0x7fa1040b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57214/0 (socket says 172.21.15.161:57214) 2024-03-23T14:35:24.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa103fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa10415e400 0x7fa1040cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33416/0 (socket says 172.21.15.161:33416) 2024-03-23T14:35:24.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa1089e4640 1 -- 172.21.15.161:0/2807720208 learned_addr learned my addr 172.21.15.161:0/2807720208 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:24.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa103fff640 1 -- 172.21.15.161:0/2807720208 >> v1:172.21.15.183:6789/0 conn(0x7fa1040cf9d0 legacy=0x7fa1040d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa103fff640 1 -- 172.21.15.161:0/2807720208 >> v1:172.21.15.161:6789/0 conn(0x7fa1040d2b40 legacy=0x7fa1040b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa103fff640 1 -- 172.21.15.161:0/2807720208 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1040d0990 con 0x7fa10415e400 2024-03-23T14:35:24.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.313+0000 7fa102ffd640 1 -- 172.21.15.161:0/2807720208 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2211077480 0 0) 0x7fa1040d0460 con 0x7fa1040d2b40 2024-03-23T14:35:24.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.314+0000 7fa103fff640 1 --2- 172.21.15.161:0/2807720208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa10415e400 0x7fa1040cf490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40097a0 tx=0x7fa0f402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85b71f9fdc20174e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.314+0000 7fa102ffd640 1 -- 172.21.15.161:0/2807720208 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4030610 con 0x7fa10415e400 2024-03-23T14:35:24.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.314+0000 7fa102ffd640 1 -- 172.21.15.161:0/2807720208 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f4030770 con 0x7fa10415e400 2024-03-23T14:35:24.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.314+0000 7fa102ffd640 1 -- 172.21.15.161:0/2807720208 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4030b00 con 0x7fa10415e400 2024-03-23T14:35:24.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.315+0000 7fa10a447640 1 -- 172.21.15.161:0/2807720208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa10415e400 msgr2=0x7fa1040cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.315+0000 7fa10a447640 1 --2- 172.21.15.161:0/2807720208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa10415e400 0x7fa1040cf490 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40097a0 tx=0x7fa0f402b960 comp rx=0 tx=0).stop 2024-03-23T14:35:24.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.315+0000 7fa10a447640 1 -- 172.21.15.161:0/2807720208 shutdown_connections 2024-03-23T14:35:24.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.315+0000 7fa10a447640 1 --2- 172.21.15.161:0/2807720208 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa10415e400 0x7fa1040cf490 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.315+0000 7fa10a447640 1 -- 172.21.15.161:0/2807720208 >> 172.21.15.161:0/2807720208 conn(0x7fa10409d190 msgr2=0x7fa1040d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:24.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.315+0000 7fa10a447640 1 -- 172.21.15.161:0/2807720208 shutdown_connections 2024-03-23T14:35:24.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.315+0000 7fa10a447640 1 -- 172.21.15.161:0/2807720208 wait complete. 2024-03-23T14:35:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.316+0000 7fa10a447640 1 Processor -- start 2024-03-23T14:35:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.316+0000 7fa10a447640 1 -- start start 2024-03-23T14:35:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa10a447640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1040cf9d0 0x7fa1041532e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa10a447640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1040d2b40 0x7fa1040caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa10a447640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa10415e400 0x7fa1040b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa10a447640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa1040cb920 con 0x7fa1040cf9d0 2024-03-23T14:35:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa10a447640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa1041f81f0 con 0x7fa1040d2b40 2024-03-23T14:35:24.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa10a447640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa1040cbb70 con 0x7fa10415e400 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1040cf9d0 0x7fa1041532e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1040cf9d0 0x7fa1041532e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33428/0 (socket says 172.21.15.161:33428) 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 -- 172.21.15.161:0/2908773335 learned_addr learned my addr 172.21.15.161:0/2908773335 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa1037fe640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1040d2b40 0x7fa1040caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa1089e4640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa10415e400 0x7fa1040b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 -- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa10415e400 msgr2=0x7fa1040b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa10415e400 0x7fa1040b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 -- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1040d2b40 msgr2=0x7fa1040caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1040d2b40 0x7fa1040caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 -- 172.21.15.161:0/2908773335 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0f4009400 con 0x7fa1040cf9d0 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa103fff640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1040cf9d0 0x7fa1041532e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa0f402bf70 tx=0x7fa0f40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa100ff9640 1 -- 172.21.15.161:0/2908773335 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f403e070 con 0x7fa1040cf9d0 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa100ff9640 1 -- 172.21.15.161:0/2908773335 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f4039a10 con 0x7fa1040cf9d0 2024-03-23T14:35:24.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa10a447640 1 -- 172.21.15.161:0/2908773335 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1040cbe50 con 0x7fa1040cf9d0 2024-03-23T14:35:24.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa100ff9640 1 -- 172.21.15.161:0/2908773335 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4039da0 con 0x7fa1040cf9d0 2024-03-23T14:35:24.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.317+0000 7fa10a447640 1 -- 172.21.15.161:0/2908773335 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1040cc410 con 0x7fa1040cf9d0 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.319+0000 7fa100ff9640 1 -- 172.21.15.161:0/2908773335 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa0f404b400 con 0x7fa1040cf9d0 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.319+0000 7fa100ff9640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa0f006b810 0x7fa0f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.319+0000 7fa100ff9640 1 -- 172.21.15.161:0/2908773335 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa0f40ac5b0 con 0x7fa1040cf9d0 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.319+0000 7fa1037fe640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa0f006b810 0x7fa0f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.320+0000 7fa1037fe640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa0f006b810 0x7fa0f006dcd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa104153cb0 tx=0x7fa0f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.320+0000 7fa10d022740 1 Processor -- start 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.320+0000 7fa10d022740 1 -- start start 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa10d022740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a610 0x5569536ed5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa10d022740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 0x55695371d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa10d022740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536edaf0 0x55695371d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa10d022740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556953661180 con 0x5569536edaf0 2024-03-23T14:35:24.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa10d022740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55695365e6b0 con 0x55695371a610 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa10d022740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x556953659e90 con 0x55695371a9f0 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa103fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a610 0x5569536ed5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa1037fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 0x55695371d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa1089e4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536edaf0 0x55695371d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa103fff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a610 0x5569536ed5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:44968/0 (socket says 172.21.15.161:44968) 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa1089e4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536edaf0 0x55695371d8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33438/0 (socket says 172.21.15.161:33438) 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa103fff640 1 -- 172.21.15.161:0/4127004921 learned_addr learned my addr 172.21.15.161:0/4127004921 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa103fff640 1 -- 172.21.15.161:0/4127004921 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 msgr2=0x55695371d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa103fff640 1 --2- 172.21.15.161:0/4127004921 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 0x55695371d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa103fff640 1 -- 172.21.15.161:0/4127004921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536edaf0 msgr2=0x55695371d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa103fff640 1 --2- 172.21.15.161:0/4127004921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536edaf0 0x55695371d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.321+0000 7fa103fff640 1 -- 172.21.15.161:0/4127004921 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556953616d80 con 0x55695371a610 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa103fff640 1 --2- 172.21.15.161:0/4127004921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a610 0x5569536ed5b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40830c0 tx=0x7fa0f407b5e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=24a48f02ccc2960 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa0e9ffb640 1 -- 172.21.15.161:0/4127004921 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f408eea0 con 0x55695371a610 2024-03-23T14:35:24.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa0e9ffb640 1 -- 172.21.15.161:0/4127004921 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f40835f0 con 0x55695371a610 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa0e9ffb640 1 -- 172.21.15.161:0/4127004921 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f4077550 con 0x55695371a610 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 -- 172.21.15.161:0/4127004921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a610 msgr2=0x5569536ed5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 --2- 172.21.15.161:0/4127004921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a610 0x5569536ed5b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40830c0 tx=0x7fa0f407b5e0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 -- 172.21.15.161:0/4127004921 shutdown_connections 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 --2- 172.21.15.161:0/4127004921 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536edaf0 0x55695371d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 --2- 172.21.15.161:0/4127004921 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 0x55695371d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 --2- 172.21.15.161:0/4127004921 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a610 0x5569536ed5b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 -- 172.21.15.161:0/4127004921 >> 172.21.15.161:0/4127004921 conn(0x5569536ef820 msgr2=0x5569537383a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 -- 172.21.15.161:0/4127004921 shutdown_connections 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.322+0000 7fa10d022740 1 -- 172.21.15.161:0/4127004921 wait complete. 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.323+0000 7fa10d022740 1 -- start start 2024-03-23T14:35:24.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.323+0000 7fa10d022740 1 Processor -- start 2024-03-23T14:35:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa10d022740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55695371a610 0x556953715550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa10d022740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 0x5569537205f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa10d022740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371c600 0x556953720b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa10d022740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5569535fed50 con 0x55695371a610 2024-03-23T14:35:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa10d022740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x55695332d310 con 0x55695371c600 2024-03-23T14:35:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa10d022740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x5569535df8f0 con 0x55695371a9f0 2024-03-23T14:35:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa10d022740 10 client.0 _ll_register_callbacks cb 0x55695336aac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T14:35:24.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55695371a610 0x556953715550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55695371a610 0x556953715550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33448/0 (socket says 172.21.15.161:33448) 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 -- 172.21.15.161:0/4172576837 learned_addr learned my addr 172.21.15.161:0/4172576837 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa1037fe640 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 0x5569537205f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa1089e4640 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371c600 0x556953720b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 -- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 msgr2=0x5569537205f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 0x5569537205f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 -- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371c600 msgr2=0x556953720b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371c600 0x556953720b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 -- 172.21.15.161:0/4172576837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1040cc410 con 0x55695371a610 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa103fff640 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55695371a610 0x556953715550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa0f407b790 tx=0x7fa0f402be80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edf5552d5384ade0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa0e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/4172576837 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f408eea0 con 0x55695371a610 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/4172576837 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa0f4081270 con 0x55695371a610 2024-03-23T14:35:24.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.324+0000 7fa10d022740 1 -- 172.21.15.161:0/4172576837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5569533596c0 con 0x55695371a610 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/4172576837 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fa0f407a330 con 0x55695371a610 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa10d022740 10 client.4626 fetch_fsmap learned FSMap version 6 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa10d022740 1 -- 172.21.15.161:0/4172576837 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x556953658510 con 0x55695371a610 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/4172576837 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x5569533596c0 con 0x55695371a610 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/4172576837 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 6) ==== 3494+0+0 (secure 0 0 0) 0x7fa0f4049020 con 0x55695371a610 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa10d022740 10 client.4626 fetch_fsmap finished waiting for FSMap version 6 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa10d022740 10 client.4626 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa10d022740 20 client.4626 populate_metadata read hostname 'smithi161' 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa10d022740 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] conn(0x5569537484e0 0x5569537552d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa10d022740 4 client.4626 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa10d022740 1 -- 172.21.15.161:0/4172576837 --> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5569533aa270 con 0x5569537484e0 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa1037fe640 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] conn(0x5569537484e0 0x5569537552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa0e2ffd640 10 client.4626 ms_handle_connect on v2:172.21.15.183:6835/1031097819 2024-03-23T14:35:24.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.325+0000 7fa1037fe640 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] conn(0x5569537484e0 0x5569537552d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=810794cbad577538 server_cookie=75c107f1d0dee2fc in_seq=0 out_seq=0 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/4172576837 <== mds.0 v2:172.21.15.183:6835/1031097819 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x5569533aa270 con 0x5569537484e0 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa0e2ffd640 10 client.4626 handle_command_reply: tid=0 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 client.4626 shutdown 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 -- 172.21.15.161:0/4172576837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55695371a610 msgr2=0x556953715550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55695371a610 0x556953715550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa0f407b790 tx=0x7fa0f402be80 comp rx=0 tx=0).stop 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 -- 172.21.15.161:0/4172576837 shutdown_connections 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371c600 0x556953720b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371a9f0 0x5569537205f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] conn(0x5569537484e0 0x5569537552d0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 --2- 172.21.15.161:0/4172576837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x55695371a610 0x556953715550 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 -- 172.21.15.161:0/4172576837 >> 172.21.15.161:0/4172576837 conn(0x5569536edaf0 msgr2=0x5569536f24b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa0e2ffd640 0 client.4626 ms_handle_reset on v2:172.21.15.183:6835/1031097819 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 -- 172.21.15.161:0/4172576837 shutdown_connections 2024-03-23T14:35:24.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 1 -- 172.21.15.161:0/4172576837 wait complete. 2024-03-23T14:35:24.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.326+0000 7fa10d022740 20 client.4626 trim_cache size 0 max 16384 2024-03-23T14:35:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.344+0000 7fa10d022740 1 Processor -- start 2024-03-23T14:35:24.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.344+0000 7fa10d022740 1 -- start start 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa10d022740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 0x556953719f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa10d022740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 0x55695373bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa10d022740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x55695375a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa10d022740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x5569536616f0 con 0x5569536ee880 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa10d022740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x5569536f12b0 con 0x55695371a9f0 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa10d022740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x5569536ef4a0 con 0x55695371c600 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1089e4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x55695375a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1089e4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x55695375a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33462/0 (socket says 172.21.15.161:33462) 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1089e4640 1 -- 172.21.15.161:0/4232341977 learned_addr learned my addr 172.21.15.161:0/4232341977 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:24.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa103fff640 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 0x556953719f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1037fe640 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 0x55695373bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1089e4640 1 -- 172.21.15.161:0/4232341977 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 msgr2=0x55695373bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1089e4640 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 0x55695373bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1089e4640 1 -- 172.21.15.161:0/4232341977 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 msgr2=0x556953719f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1089e4640 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 0x556953719f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.345+0000 7fa1089e4640 1 -- 172.21.15.161:0/4232341977 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5569535f4a80 con 0x5569536ee880 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa1089e4640 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x55695375a520 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc005fd0 tx=0x7fa0fc007710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b2ef7cf6464349c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa0e97fa640 1 -- 172.21.15.161:0/4232341977 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc007a20 con 0x5569536ee880 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa0e97fa640 1 -- 172.21.15.161:0/4232341977 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0fc001d70 con 0x5569536ee880 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa0e97fa640 1 -- 172.21.15.161:0/4232341977 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0fc001f00 con 0x5569536ee880 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 -- 172.21.15.161:0/4232341977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 msgr2=0x55695375a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x55695375a520 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fa0fc005fd0 tx=0x7fa0fc007710 comp rx=0 tx=0).stop 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 -- 172.21.15.161:0/4232341977 shutdown_connections 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x55695375a520 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 0x55695373bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 --2- 172.21.15.161:0/4232341977 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 0x556953719f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 -- 172.21.15.161:0/4232341977 >> 172.21.15.161:0/4232341977 conn(0x5569536edaf0 msgr2=0x55695371add0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:24.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 -- 172.21.15.161:0/4232341977 shutdown_connections 2024-03-23T14:35:24.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.346+0000 7fa10d022740 1 -- 172.21.15.161:0/4232341977 wait complete. 2024-03-23T14:35:24.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.347+0000 7fa10d022740 1 -- start start 2024-03-23T14:35:24.345 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.347+0000 7fa10d022740 1 Processor -- start 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa10d022740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x5569537b0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa10d022740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 0x556953714080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa10d022740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 0x5569537471f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa10d022740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x556953746dd0 con 0x5569536ee880 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa10d022740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x556953746ac0 con 0x55695371a9f0 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa10d022740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x5569537145c0 con 0x55695371c600 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa10d022740 10 client.0 _ll_register_callbacks cb 0x7fa0f4091070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x5569537b0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x5569537b0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33466/0 (socket says 172.21.15.161:33466) 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 -- 172.21.15.161:0/3944002060 learned_addr learned my addr 172.21.15.161:0/3944002060 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa1037fe640 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 0x556953714080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa1089e4640 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 0x5569537471f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 -- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 msgr2=0x5569537471f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 0x5569537471f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 -- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 msgr2=0x556953714080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 0x556953714080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 -- 172.21.15.161:0/3944002060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0fc00f040 con 0x5569536ee880 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa103fff640 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x5569537b0890 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40097a0 tx=0x7fa0f404c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=64e4cabbbde23c6f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa0e2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/3944002060 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f408eea0 con 0x5569536ee880 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/3944002060 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa0f407a640 con 0x5569536ee880 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.348+0000 7fa10d022740 1 -- 172.21.15.161:0/3944002060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0f40835f0 con 0x5569536ee880 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/3944002060 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=6) ==== 24+0+0 (secure 0 0 0) 0x7fa0f4081320 con 0x5569536ee880 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa10d022740 10 client.4632 fetch_fsmap learned FSMap version 6 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa10d022740 1 -- 172.21.15.161:0/3944002060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=6,monmap=2+}) -- 0x55695361a7e0 con 0x5569536ee880 2024-03-23T14:35:24.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/3944002060 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0f40835f0 con 0x5569536ee880 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/3944002060 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 6) ==== 3494+0+0 (secure 0 0 0) 0x7fa0f4076c30 con 0x5569536ee880 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa10d022740 10 client.4632 fetch_fsmap finished waiting for FSMap version 6 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa10d022740 10 client.4632 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa10d022740 20 client.4632 populate_metadata read hostname 'smithi161' 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa10d022740 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] conn(0x55695371add0 0x55695371b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa10d022740 4 client.4632 mds_command: new command op to 4282 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa10d022740 1 -- 172.21.15.161:0/3944002060 --> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] -- command(tid 0: {"prefix": "status"}) -- 0x556953396610 con 0x55695371add0 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa1037fe640 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] conn(0x55695371add0 0x55695371b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa1037fe640 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] conn(0x55695371add0 0x55695371b1b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b712f3139de048d server_cookie=7ab9433146f186f0 in_seq=0 out_seq=0 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.349+0000 7fa0e2ffd640 10 client.4632 ms_handle_connect on v2:172.21.15.183:6835/1031097819 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa0e2ffd640 1 -- 172.21.15.161:0/3944002060 <== mds.0 v2:172.21.15.183:6835/1031097819 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x556953396610 con 0x55695371add0 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa0e2ffd640 10 client.4632 handle_command_reply: tid=0 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 client.4632 shutdown 2024-03-23T14:35:24.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 -- 172.21.15.161:0/3944002060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 msgr2=0x5569537b0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x5569537b0890 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa0f40097a0 tx=0x7fa0f404c000 comp rx=0 tx=0).stop 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 -- 172.21.15.161:0/3944002060 shutdown_connections 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x55695371c600 0x5569537471f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x55695371a9f0 0x556953714080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.183:6835/1031097819,v1:172.21.15.183:6837/1031097819] conn(0x55695371add0 0x55695371b1b0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 --2- 172.21.15.161:0/3944002060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x5569536ee880 0x5569537b0890 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa0e2ffd640 0 client.4632 ms_handle_reset on v2:172.21.15.183:6835/1031097819 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 -- 172.21.15.161:0/3944002060 >> 172.21.15.161:0/3944002060 conn(0x5569536edaf0 msgr2=0x55695371be00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 -- 172.21.15.161:0/3944002060 shutdown_connections 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 1 -- 172.21.15.161:0/3944002060 wait complete. 2024-03-23T14:35:24.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10d022740 20 client.4632 trim_cache size 0 max 16384 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "9a16b48f-76af-4198-8fe3-d2c20a776a3e", 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4282, 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 10.593043304, 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 6, 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 20, 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 20, 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 17.625515890999999 2024-03-23T14:35:24.353 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:35:24.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 -- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa0f006b810 msgr2=0x7fa0f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa0f006b810 0x7fa0f006dcd0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa104153cb0 tx=0x7fa0f8009040 comp rx=0 tx=0).stop 2024-03-23T14:35:24.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 -- 172.21.15.161:0/2908773335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1040cf9d0 msgr2=0x7fa1041532e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1040cf9d0 0x7fa1041532e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa0f402bf70 tx=0x7fa0f40046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 -- 172.21.15.161:0/2908773335 shutdown_connections 2024-03-23T14:35:24.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa0f006b810 0x7fa0f006dcd0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa10415e400 0x7fa1040b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1040d2b40 0x7fa1040caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 --2- 172.21.15.161:0/2908773335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1040cf9d0 0x7fa1041532e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.350+0000 7fa10a447640 1 -- 172.21.15.161:0/2908773335 >> 172.21.15.161:0/2908773335 conn(0x7fa10409d190 msgr2=0x7fa104161a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:24.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.351+0000 7fa10a447640 1 -- 172.21.15.161:0/2908773335 shutdown_connections 2024-03-23T14:35:24.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.351+0000 7fa10a447640 1 -- 172.21.15.161:0/2908773335 wait complete. 2024-03-23T14:35:24.357 INFO:teuthology.run_tasks:Running task sequential... 2024-03-23T14:35:24.366 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-03-23T14:35:24.366 INFO:teuthology.task.sequential:In sequential, running task print... 2024-03-23T14:35:24.367 INFO:teuthology.task.print:Enabling mgr modules 2024-03-23T14:35:24.367 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-03-23T14:35:24.427 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-03-23T14:35:24.427 INFO:tasks.ceph_fuse:config is {'client.0': {}, 'client.2': {}, 'client.1': {}, 'client.3': {}} 2024-03-23T14:35:24.427 INFO:tasks.ceph_fuse:client.0 config is {} 2024-03-23T14:35:24.427 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T14:35:24.427 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:24.427 INFO:tasks.ceph_fuse:client.2 config is {} 2024-03-23T14:35:24.427 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T14:35:24.428 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-23T14:35:24.428 INFO:tasks.ceph_fuse:client.1 config is {} 2024-03-23T14:35:24.428 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T14:35:24.428 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:24.428 INFO:tasks.ceph_fuse:client.3 config is {} 2024-03-23T14:35:24.428 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-23T14:35:24.428 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-23T14:35:24.428 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:24.428 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:35:24.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:24.444 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link delete ceph-brx 2024-03-23T14:35:24.512 INFO:teuthology.orchestra.run.smithi161.stderr:Cannot find device "ceph-brx" 2024-03-23T14:35:24.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:35:24.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:24.514 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-03-23T14:35:24.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:24.530 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link delete ceph-brx 2024-03-23T14:35:24.603 INFO:teuthology.orchestra.run.smithi183.stderr:Cannot find device "ceph-brx" 2024-03-23T14:35:24.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:35:24.604 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-03-23T14:35:24.605 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.709+0000 7f75c1c8f640 1 Processor -- start 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.709+0000 7f75c1c8f640 1 -- start start 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.710+0000 7f75c1c8f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75bc15e400 0x7f75bc0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.710+0000 7f75c1c8f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75bc0d0460 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.710+0000 7f75c1c8f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f75bc0d06b0 con 0x7f75bc0d2b40 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.710+0000 7f75c1c8f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f75bc0d1050 con 0x7f75bc15e400 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.710+0000 7f75bb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75bc15e400 0x7f75bc0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.710+0000 7f75bb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f75bc15e400 0x7f75bc0cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33482/0 (socket says 172.21.15.161:33482) 2024-03-23T14:35:24.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.710+0000 7f75baffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f75bc0cf9d0 0x7f75bc0d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57220/0 (socket says 172.21.15.161:57220) 2024-03-23T14:35:24.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.710+0000 7f75bb7fe640 1 -- 172.21.15.161:0/1527540818 learned_addr learned my addr 172.21.15.161:0/1527540818 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:24.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.711+0000 7f75bb7fe640 1 -- 172.21.15.161:0/1527540818 >> v1:172.21.15.183:6789/0 conn(0x7f75bc0d2b40 legacy=0x7f75bc0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.711+0000 7f75bb7fe640 1 -- 172.21.15.161:0/1527540818 >> v1:172.21.15.161:6789/0 conn(0x7f75bc0cf9d0 legacy=0x7f75bc0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.711+0000 7f75ba7fc640 1 -- 172.21.15.161:0/1527540818 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2839547180 0 0) 0x7f75bc0d06b0 con 0x7f75bc0d2b40 2024-03-23T14:35:24.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.711+0000 7f75bb7fe640 1 -- 172.21.15.161:0/1527540818 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75bc0d0990 con 0x7f75bc15e400 2024-03-23T14:35:24.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.711+0000 7f75bb7fe640 1 --2- 172.21.15.161:0/1527540818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc15e400 0x7f75bc0cf490 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009870 tx=0x7f75ac02b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66bd5a3117e4d6cb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75ba7fc640 1 -- 172.21.15.161:0/1527540818 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac031740 con 0x7f75bc15e400 2024-03-23T14:35:24.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75ba7fc640 1 -- 172.21.15.161:0/1527540818 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ac0318a0 con 0x7f75bc15e400 2024-03-23T14:35:24.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75ba7fc640 1 -- 172.21.15.161:0/1527540818 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac031c30 con 0x7f75bc15e400 2024-03-23T14:35:24.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75c1c8f640 1 -- 172.21.15.161:0/1527540818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc15e400 msgr2=0x7f75bc0cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75c1c8f640 1 --2- 172.21.15.161:0/1527540818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc15e400 0x7f75bc0cf490 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f75ac009870 tx=0x7f75ac02b860 comp rx=0 tx=0).stop 2024-03-23T14:35:24.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75c1c8f640 1 -- 172.21.15.161:0/1527540818 shutdown_connections 2024-03-23T14:35:24.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75c1c8f640 1 --2- 172.21.15.161:0/1527540818 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc15e400 0x7f75bc0cf490 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75c1c8f640 1 -- 172.21.15.161:0/1527540818 >> 172.21.15.161:0/1527540818 conn(0x7f75bc09d190 msgr2=0x7f75bc0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:24.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75c1c8f640 1 -- 172.21.15.161:0/1527540818 shutdown_connections 2024-03-23T14:35:24.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.712+0000 7f75c1c8f640 1 -- 172.21.15.161:0/1527540818 wait complete. 2024-03-23T14:35:24.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.713+0000 7f75c1c8f640 1 Processor -- start 2024-03-23T14:35:24.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75c1c8f640 1 -- start start 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75c1c8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc0cf9d0 0x7f75bc0b8960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75c1c8f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75bc0d2b40 0x7f75bc1f7b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75c1c8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75bc15e400 0x7f75bc161170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75c1c8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc0c89d0 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75c1c8f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f75bc1f80b0 con 0x7f75bc0d2b40 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75c1c8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f75bc0b9060 con 0x7f75bc15e400 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75bb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc0cf9d0 0x7f75bc0b8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75bb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc0cf9d0 0x7f75bc0b8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33492/0 (socket says 172.21.15.161:33492) 2024-03-23T14:35:24.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75bb7fe640 1 -- 172.21.15.161:0/896193637 learned_addr learned my addr 172.21.15.161:0/896193637 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75baffd640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75bc0d2b40 0x7f75bc1f7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.714+0000 7f75bbfff640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75bc15e400 0x7f75bc161170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75bb7fe640 1 -- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75bc15e400 msgr2=0x7f75bc161170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75bb7fe640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75bc15e400 0x7f75bc161170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75bb7fe640 1 -- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75bc0d2b40 msgr2=0x7f75bc1f7b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75bb7fe640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75bc0d2b40 0x7f75bc1f7b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75bb7fe640 1 -- 172.21.15.161:0/896193637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75ac0094d0 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75bb7fe640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc0cf9d0 0x7f75bc0b8960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f75ac000c00 tx=0x7f75ac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75b3fff640 1 -- 172.21.15.161:0/896193637 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac033c70 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75b3fff640 1 -- 172.21.15.161:0/896193637 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f75ac033dd0 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75b3fff640 1 -- 172.21.15.161:0/896193637 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f75ac03c640 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75bc0b9340 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.715+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75bc0b9900 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.716+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75bc068680 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.717+0000 7f75b3fff640 1 -- 172.21.15.161:0/896193637 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f75ac03a040 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.717+0000 7f75b3fff640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f759c06b810 0x7f759c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:24.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.717+0000 7f75b3fff640 1 -- 172.21.15.161:0/896193637 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f75ac0abc40 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.717+0000 7f75baffd640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f759c06b810 0x7f759c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:24.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.717+0000 7f75baffd640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f759c06b810 0x7f759c06dcd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0a1dd0 tx=0x7f75a4007e00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:24.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.721+0000 7f75b3fff640 1 -- 172.21.15.161:0/896193637 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f75ac0767a0 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.931+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f75bc0b9af0 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.932+0000 7f75b3fff640 1 -- 172.21.15.161:0/896193637 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+73 (secure 0 0 0) 0x7f75ac04d070 con 0x7f75bc0cf9d0 2024-03-23T14:35:24.928 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.933+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f759c06b810 msgr2=0x7f759c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.933+0000 7f75c1c8f640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f759c06b810 0x7f759c06dcd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0a1dd0 tx=0x7f75a4007e00 comp rx=0 tx=0).stop 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.933+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc0cf9d0 msgr2=0x7f75bc0b8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.933+0000 7f75c1c8f640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc0cf9d0 0x7f75bc0b8960 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f75ac000c00 tx=0x7f75ac0046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.934+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 shutdown_connections 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.934+0000 7f75c1c8f640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f759c06b810 0x7f759c06dcd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.934+0000 7f75c1c8f640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f75bc15e400 0x7f75bc161170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.934+0000 7f75c1c8f640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f75bc0d2b40 0x7f75bc1f7b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.934+0000 7f75c1c8f640 1 --2- 172.21.15.161:0/896193637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f75bc0cf9d0 0x7f75bc0b8960 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:24.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.934+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 >> 172.21.15.161:0/896193637 conn(0x7f75bc09d190 msgr2=0x7f75bc0b5940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:24.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.934+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 shutdown_connections 2024-03-23T14:35:24.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:24.934+0000 7f75c1c8f640 1 -- 172.21.15.161:0/896193637 wait complete. 2024-03-23T14:35:24.940 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:35:24.940 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:35:24.940 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-23T14:35:24.940 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T14:35:24.940 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:24.941 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:35:24.941 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:35:24.941 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:35:24.941 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:35:24.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:24.941 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-23T14:35:24.956 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42745sec preferred_lft 42745sec 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-23T14:35:24.957 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:24.957 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T14:35:24.958 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:24.958 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:24.958 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link add name ceph-brx type bridge 2024-03-23T14:35:24.958 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip addr flush dev ceph-brx 2024-03-23T14:35:24.958 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set ceph-brx up 2024-03-23T14:35:24.958 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T14:35:24.958 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:25.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:25.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:25.168 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:25.168 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:35:25.195 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-23T14:35:25.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:25.197 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-23T14:35:25.265 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-23T14:35:25.265 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:35:25.265 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:35:25.266 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:35:25.266 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:35:25.266 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:25.266 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:25.266 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:35:25.266 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:35:25.266 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:35:25.266 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:25.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:25.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:25.581 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:25.581 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:35:25.596 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:25.596 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list-id 2024-03-23T14:35:25.652 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:25.652 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:25.652 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:25.652 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-23T14:35:25.652 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:25.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:25.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:25.773 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-23T14:35:25.773 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:25.774 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:25.774 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-23T14:35:25.774 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T14:35:25.774 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-23T14:35:25.774 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-23T14:35:25.774 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-23T14:35:25.774 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:25.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:25.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:25.914 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:25.914 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:25.914 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-23T14:35:25.914 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T14:35:25.914 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:25.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:25.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:25.995 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T14:35:25.995 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:35:25.996 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:26.010 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:35:26.010 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:35:26.011 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:26.066 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:35:26.134 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:35:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:35:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:35:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:35:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:35:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:35:26.192 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:26.192 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:26.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:26.192 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:35:26.244 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T14:35:26.244 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T14:35:26.287 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:35:26.313 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:35:26.369 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:35:26.369 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:35:26.369 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:35:26.369 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:35:26.369 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:35:26.369 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:35:26.369 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:35:26.369 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:35:26.370 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:26.371 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:26.372 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:35:26.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:2024-03-23T14:35:26.377+0000 7f2306266480 -1 init, newargv = 0x55f7fc30ea70 newargc=15 2024-03-23T14:35:26.373 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[45801]: starting ceph client 2024-03-23T14:35:26.377 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[45801]: starting fuse 2024-03-23T14:35:26.388 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:35:26.388 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-23T14:35:26.388 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T14:35:26.389 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-23T14:35:26.390 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-23T14:35:26.391 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:35:26.391 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:35:26.391 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.0")) 2024-03-23T14:35:26.391 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:35:26.508 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.0.45801.asok 2024-03-23T14:35:26.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:26.518 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.45801.asok 2024-03-23T14:35:26.518 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:26.518 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.45801.asok status 2024-03-23T14:35:26.655 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:35:26.655 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-23T14:35:26.655 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:35:26.655 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:35:26.655 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "0", 2024-03-23T14:35:26.655 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-23T14:35:26.655 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T14:35:26.655 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "45801", 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4647, 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 4647 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:35:26.656 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 4201418851 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 4201418851 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.4647 192.168.0.1:0/4201418851", 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/4201418851", 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 6, 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 20, 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-23T14:35:26.657 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-23T14:35:26.658 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:35:26.663 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:35:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.769+0000 7fa370803640 1 Processor -- start 2024-03-23T14:35:26.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.769+0000 7fa370803640 1 -- start start 2024-03-23T14:35:26.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa370803640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15b720 0x7fa36c0cd740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:26.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa370803640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa36c0d1880 0x7fa36c0d1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:26.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa370803640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa36c0cd2b0 con 0x7fa36c15bb00 2024-03-23T14:35:26.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa370803640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa36c0cc720 con 0x7fa36c0d1880 2024-03-23T14:35:26.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa370803640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa36c0d2270 con 0x7fa36c15b720 2024-03-23T14:35:26.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa36ad9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15b720 0x7fa36c0cd740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:26.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa36b59e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa36c0d1880 0x7fa36c0d1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:26.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa36ad9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15b720 0x7fa36c0cd740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:56016/0 (socket says 172.21.15.161:56016) 2024-03-23T14:35:26.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa36ad9d640 1 -- 172.21.15.161:0/3102521233 learned_addr learned my addr 172.21.15.161:0/3102521233 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:26.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa36b59e640 1 -- 172.21.15.161:0/3102521233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15b720 msgr2=0x7fa36c0cd740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:26.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa36b59e640 1 --2- 172.21.15.161:0/3102521233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15b720 0x7fa36c0cd740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa36b59e640 1 -- 172.21.15.161:0/3102521233 >> v1:172.21.15.183:6789/0 conn(0x7fa36c15bb00 legacy=0x7fa36c15eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:26.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.770+0000 7fa36b59e640 1 -- 172.21.15.161:0/3102521233 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa36c0cf340 con 0x7fa36c0d1880 2024-03-23T14:35:26.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.771+0000 7fa36b59e640 1 --2- 172.21.15.161:0/3102521233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c0d1d30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa364009d90 tx=0x7fa36402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f917f0be4120e436 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:26.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.771+0000 7fa36a59c640 1 -- 172.21.15.161:0/3102521233 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa36402cc50 con 0x7fa36c0d1880 2024-03-23T14:35:26.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.771+0000 7fa36a59c640 1 -- 172.21.15.161:0/3102521233 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa36402cdb0 con 0x7fa36c0d1880 2024-03-23T14:35:26.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.771+0000 7fa36a59c640 1 -- 172.21.15.161:0/3102521233 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa364034660 con 0x7fa36c0d1880 2024-03-23T14:35:26.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.772+0000 7fa370803640 1 -- 172.21.15.161:0/3102521233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 msgr2=0x7fa36c0d1d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:26.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.772+0000 7fa370803640 1 --2- 172.21.15.161:0/3102521233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c0d1d30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fa364009d90 tx=0x7fa36402bea0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.772+0000 7fa370803640 1 -- 172.21.15.161:0/3102521233 shutdown_connections 2024-03-23T14:35:26.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.772+0000 7fa370803640 1 --2- 172.21.15.161:0/3102521233 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c0d1d30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.772+0000 7fa370803640 1 --2- 172.21.15.161:0/3102521233 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15b720 0x7fa36c0cd740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.772+0000 7fa370803640 1 -- 172.21.15.161:0/3102521233 >> 172.21.15.161:0/3102521233 conn(0x7fa36c09aa40 msgr2=0x7fa36c09ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:26.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.772+0000 7fa370803640 1 -- 172.21.15.161:0/3102521233 shutdown_connections 2024-03-23T14:35:26.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.772+0000 7fa370803640 1 -- 172.21.15.161:0/3102521233 wait complete. 2024-03-23T14:35:26.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.773+0000 7fa370803640 1 Processor -- start 2024-03-23T14:35:26.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.773+0000 7fa370803640 1 -- start start 2024-03-23T14:35:26.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa370803640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c1f0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:26.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa370803640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c15b720 0x7fa36c0b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:26.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa370803640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15bb00 0x7fa36c0b6150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:26.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa370803640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c0b5d90 con 0x7fa36c15b720 2024-03-23T14:35:26.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa370803640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa36c0b46d0 con 0x7fa36c0d1880 2024-03-23T14:35:26.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa370803640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa36c1f3bf0 con 0x7fa36c15bb00 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa363fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c15b720 0x7fa36c0b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa36ad9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c1f0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa36ad9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c1f0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:45010/0 (socket says 172.21.15.161:45010) 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa36ad9d640 1 -- 172.21.15.161:0/2689664237 learned_addr learned my addr 172.21.15.161:0/2689664237 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa363fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c15b720 0x7fa36c0b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33502/0 (socket says 172.21.15.161:33502) 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa36b59e640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15bb00 0x7fa36c0b6150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa363fff640 1 -- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15bb00 msgr2=0x7fa36c0b6150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa363fff640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15bb00 0x7fa36c0b6150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa363fff640 1 -- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 msgr2=0x7fa36c1f0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa363fff640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c1f0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa363fff640 1 -- 172.21.15.161:0/2689664237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa364009a70 con 0x7fa36c15b720 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.774+0000 7fa36ad9d640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c1f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.775+0000 7fa363fff640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c15b720 0x7fa36c0b5850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa35400c390 tx=0x7fa35400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:26.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.775+0000 7fa3637fe640 1 -- 172.21.15.161:0/2689664237 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3540052d0 con 0x7fa36c15b720 2024-03-23T14:35:26.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.775+0000 7fa3637fe640 1 -- 172.21.15.161:0/2689664237 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa354005450 con 0x7fa36c15b720 2024-03-23T14:35:26.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.775+0000 7fa3637fe640 1 -- 172.21.15.161:0/2689664237 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa354007260 con 0x7fa36c15b720 2024-03-23T14:35:26.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.775+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa36c1f3ed0 con 0x7fa36c15b720 2024-03-23T14:35:26.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.775+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa36c1f4490 con 0x7fa36c15b720 2024-03-23T14:35:26.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.776+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa36c04f470 con 0x7fa36c15b720 2024-03-23T14:35:26.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.776+0000 7fa3637fe640 1 -- 172.21.15.161:0/2689664237 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa3540073c0 con 0x7fa36c15b720 2024-03-23T14:35:26.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.777+0000 7fa3637fe640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa34c06b810 0x7fa34c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:26.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.777+0000 7fa3637fe640 1 -- 172.21.15.161:0/2689664237 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa3540892d0 con 0x7fa36c15b720 2024-03-23T14:35:26.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.777+0000 7fa36ad9d640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa34c06b810 0x7fa34c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:26.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.777+0000 7fa36ad9d640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa34c06b810 0x7fa34c06dcd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0094a0 tx=0x7fa35c009240 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:26.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.781+0000 7fa3637fe640 1 -- 172.21.15.161:0/2689664237 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa354053e30 con 0x7fa36c15b720 2024-03-23T14:35:26.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.990+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fa36c1f4650 con 0x7fa36c15b720 2024-03-23T14:35:26.986 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.991+0000 7fa3637fe640 1 -- 172.21.15.161:0/2689664237 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+73 (secure 0 0 0) 0x7fa3540574f0 con 0x7fa36c15b720 2024-03-23T14:35:26.987 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa34c06b810 msgr2=0x7fa34c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa34c06b810 0x7fa34c06dcd0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa35c0094a0 tx=0x7fa35c009240 comp rx=0 tx=0).stop 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c15b720 msgr2=0x7fa36c0b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c15b720 0x7fa36c0b5850 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa35400c390 tx=0x7fa35400c850 comp rx=0 tx=0).stop 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 shutdown_connections 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa34c06b810 0x7fa34c06dcd0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa36c15bb00 0x7fa36c0b6150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa36c15b720 0x7fa36c0b5850 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.989 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 --2- 172.21.15.161:0/2689664237 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa36c0d1880 0x7fa36c1f0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:26.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 >> 172.21.15.161:0/2689664237 conn(0x7fa36c09aa40 msgr2=0x7fa36c0b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:26.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 shutdown_connections 2024-03-23T14:35:26.990 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:26.993+0000 7fa370803640 1 -- 172.21.15.161:0/2689664237 wait complete. 2024-03-23T14:35:26.999 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:35:26.999 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:35:26.999 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-23T14:35:27.000 INFO:tasks.cephfs.mount:self.client.name = client.2 2024-03-23T14:35:27.000 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.2 2024-03-23T14:35:27.000 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:35:27.000 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:35:27.000 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:35:27.000 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.2' 2024-03-23T14:35:27.000 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:27.000 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:27.016 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42743sec preferred_lft 42743sec 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:35:27.017 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c4a9:64ff:fe59:877a/64 scope link 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run.smithi161.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c48:50ff:fe01:9f4e/64 scope link tentative 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.018 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:27.018 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:35:27.082 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-23T14:35:27.083 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:27.083 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-23T14:35:27.139 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-23T14:35:27.139 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:35:27.139 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:35:27.139 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:35:27.139 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:35:27.140 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:27.140 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:27.140 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:35:27.140 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:35:27.140 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:35:27.140 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:27.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:27.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:27.274 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:27.274 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:35:27.289 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T14:35:27.289 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:27.290 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list-id 2024-03-23T14:35:27.344 INFO:teuthology.orchestra.run.smithi161.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.0) 2024-03-23T14:35:27.345 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:27.345 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:27.345 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:35:27.345 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.2 1 2024-03-23T14:35:27.345 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:27.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:27.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:27.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:27.467 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-23T14:35:27.493 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 9e:e0:66:17:7b:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::9ce0:66ff:fe17:7b30/64 scope link tentative 2024-03-23T14:35:27.494 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.495 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:27.496 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr 2024-03-23T14:35:27.564 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:35:27.564 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:35:27.564 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:35:27.564 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.564 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:35:27.564 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.564 INFO:teuthology.orchestra.run.smithi161.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:35:27.564 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 9e:e0:66:17:7b:30 brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T14:35:27.565 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T14:35:27.565 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.565 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::9ce0:66ff:fe17:7b30/64 scope link tentative 2024-03-23T14:35:27.565 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:27.565 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.2' with 192.168.0.2/16 2024-03-23T14:35:27.565 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:27.565 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:27.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.2 type veth peer name brx.1 2024-03-23T14:35:27.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-23T14:35:27.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set veth0 up 2024-03-23T14:35:27.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip link set lo up 2024-03-23T14:35:27.566 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.2 ip route add default via 192.168.255.254 2024-03-23T14:35:27.566 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:27.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:27.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:27.729 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:27.730 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:35:27.730 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.1 up 2024-03-23T14:35:27.730 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set dev brx.1 master ceph-brx 2024-03-23T14:35:27.730 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:35:27.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:27.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:27.813 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2024-03-23T14:35:27.813 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:35:27.813 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.2 2024-03-23T14:35:27.828 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.2' 2024-03-23T14:35:27.828 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:35:27.828 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.2 2024-03-23T14:35:27.883 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:35:27.951 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:35:28.007 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:35:28.007 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:35:28.007 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:35:28.007 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:35:28.008 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:35:28.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:28.010 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:28.010 INFO:teuthology.orchestra.run.smithi161.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:28.010 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:35:28.010 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:35:28.011 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:28.011 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:35:28.062 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:35:28.062 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-23T14:35:28.062 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.2 --id 2) 2024-03-23T14:35:28.105 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:35:28.132 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:35:28.188 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:35:28.188 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:35:28.189 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:35:28.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:35:28.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:35:28.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:28.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:35:28.191 INFO:teuthology.orchestra.run.smithi161.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:28.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:35:28.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:35:28.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:28.191 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:35:28.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi161.stderr:2024-03-23T14:35:28.195+0000 7f7385236480 -1 init, newargv = 0x564e8f88a650 newargc=15 2024-03-23T14:35:28.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi161.stderr:ceph-fuse[46299]: starting ceph client 2024-03-23T14:35:28.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi161.stderr:ceph-fuse[46299]: starting fuse 2024-03-23T14:35:28.206 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:35:28.207 INFO:teuthology.orchestra.run.smithi161.stdout:44 2024-03-23T14:35:28.207 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-23T14:35:28.207 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:35:28.207 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-23T14:35:28.207 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-23T14:35:28.207 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:35:28.207 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-23T14:35:28.207 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:35:28.207 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-23T14:35:28.207 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.2" 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-23T14:35:28.208 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-23T14:35:28.209 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-23T14:35:28.209 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:35:28.209 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:35:28.209 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.2")) 2024-03-23T14:35:28.209 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:35:28.327 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.2.46299.asok 2024-03-23T14:35:28.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:28.337 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.46299.asok 2024-03-23T14:35:28.337 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:28.337 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.46299.asok status 2024-03-23T14:35:28.474 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:35:28.474 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-23T14:35:28.474 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:35:28.474 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:35:28.474 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "2", 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "46299", 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 4656, 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 4656 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:35:28.475 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.2:0", 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3848420141 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.2:0", 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3848420141 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.4656 192.168.0.2:0/3848420141", 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.2:0/3848420141", 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 6, 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 20, 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:35:28.476 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-23T14:35:28.477 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-23T14:35:28.477 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:35:28.483 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:35:28.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.587+0000 7f2e833f4640 1 Processor -- start 2024-03-23T14:35:28.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.587+0000 7f2e833f4640 1 -- start start 2024-03-23T14:35:28.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e833f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:28.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e833f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e7c15be70 con 0x7f2e7c06eca0 2024-03-23T14:35:28.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e833f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e7c15c150 con 0x7f2e7c0724e0 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e833f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7c15c430 con 0x7f2e7c072100 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e81991640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2e7c06eca0 0x7f2e7c0b3470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:57232/0 (socket says 172.21.15.161:57232) 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e81991640 1 -- 172.21.15.161:0/294455574 learned_addr learned my addr 172.21.15.161:0/294455574 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e81190640 1 --2- 172.21.15.161:0/294455574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e73fff640 1 -- 172.21.15.161:0/294455574 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 444181764 0 0) 0x7f2e7c15be70 con 0x7f2e7c06eca0 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e73fff640 1 -- 172.21.15.161:0/294455574 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e68003520 con 0x7f2e7c06eca0 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e73fff640 1 -- 172.21.15.161:0/294455574 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2133645063 0 0) 0x7f2e7c15c150 con 0x7f2e7c0724e0 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e73fff640 1 -- 172.21.15.161:0/294455574 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e7c15be70 con 0x7f2e7c0724e0 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.588+0000 7f2e81190640 1 -- 172.21.15.161:0/294455574 >> v1:172.21.15.183:6789/0 conn(0x7f2e7c0724e0 legacy=0x7f2e7c06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.589+0000 7f2e81190640 1 -- 172.21.15.161:0/294455574 >> v1:172.21.15.161:6789/0 conn(0x7f2e7c06eca0 legacy=0x7f2e7c0b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.589+0000 7f2e81190640 1 -- 172.21.15.161:0/294455574 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e7c0b2c70 con 0x7f2e7c072100 2024-03-23T14:35:28.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.589+0000 7f2e81190640 1 --2- 172.21.15.161:0/294455574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c15b3f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009dc0 tx=0x7f2e7802e130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=44c36b3e605f070d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:28.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.589+0000 7f2e73fff640 1 -- 172.21.15.161:0/294455574 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78036440 con 0x7f2e7c072100 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.589+0000 7f2e73fff640 1 -- 172.21.15.161:0/294455574 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e780365a0 con 0x7f2e7c072100 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.589+0000 7f2e73fff640 1 -- 172.21.15.161:0/294455574 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78036910 con 0x7f2e7c072100 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.590+0000 7f2e833f4640 1 -- 172.21.15.161:0/294455574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 msgr2=0x7f2e7c15b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.590+0000 7f2e833f4640 1 --2- 172.21.15.161:0/294455574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c15b3f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f2e78009dc0 tx=0x7f2e7802e130 comp rx=0 tx=0).stop 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.590+0000 7f2e833f4640 1 -- 172.21.15.161:0/294455574 shutdown_connections 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.590+0000 7f2e833f4640 1 --2- 172.21.15.161:0/294455574 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c15b3f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.590+0000 7f2e833f4640 1 -- 172.21.15.161:0/294455574 >> 172.21.15.161:0/294455574 conn(0x7f2e7c0b17f0 msgr2=0x7f2e7c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.590+0000 7f2e833f4640 1 -- 172.21.15.161:0/294455574 shutdown_connections 2024-03-23T14:35:28.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.590+0000 7f2e833f4640 1 -- 172.21.15.161:0/294455574 wait complete. 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.591+0000 7f2e833f4640 1 Processor -- start 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.591+0000 7f2e833f4640 1 -- start start 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e833f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e7c06eca0 0x7f2e7c1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e833f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c1e4af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e833f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e7c0724e0 0x7f2e7c1e5030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e833f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c159a90 con 0x7f2e7c06eca0 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e833f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2e7c155bc0 con 0x7f2e7c0724e0 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e833f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2e7c155ea0 con 0x7f2e7c072100 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81190640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e7c06eca0 0x7f2e7c1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81190640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e7c06eca0 0x7f2e7c1552e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:33504/0 (socket says 172.21.15.161:33504) 2024-03-23T14:35:28.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81190640 1 -- 172.21.15.161:0/2857788601 learned_addr learned my addr 172.21.15.161:0/2857788601 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e8098f640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c1e4af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81991640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e7c0724e0 0x7f2e7c1e5030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81190640 1 -- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 msgr2=0x7f2e7c1e4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81190640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c1e4af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81190640 1 -- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e7c0724e0 msgr2=0x7f2e7c1e5030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81190640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e7c0724e0 0x7f2e7c1e5030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.592+0000 7f2e81190640 1 -- 172.21.15.161:0/2857788601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e78009a70 con 0x7f2e7c06eca0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.593+0000 7f2e81190640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e7c06eca0 0x7f2e7c1552e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2e78000f80 tx=0x7f2e78004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.593+0000 7f2e71ffb640 1 -- 172.21.15.161:0/2857788601 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78042860 con 0x7f2e7c06eca0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.593+0000 7f2e71ffb640 1 -- 172.21.15.161:0/2857788601 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e780429e0 con 0x7f2e7c06eca0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.593+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e7c156180 con 0x7f2e7c06eca0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.593+0000 7f2e71ffb640 1 -- 172.21.15.161:0/2857788601 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e78042d90 con 0x7f2e7c06eca0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.593+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e7c141740 con 0x7f2e7c06eca0 2024-03-23T14:35:28.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.594+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e7c0735e0 con 0x7f2e7c06eca0 2024-03-23T14:35:28.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.594+0000 7f2e71ffb640 1 -- 172.21.15.161:0/2857788601 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2e78040400 con 0x7f2e7c06eca0 2024-03-23T14:35:28.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.595+0000 7f2e71ffb640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e6006b810 0x7f2e6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:28.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.595+0000 7f2e71ffb640 1 -- 172.21.15.161:0/2857788601 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2e780af2c0 con 0x7f2e7c06eca0 2024-03-23T14:35:28.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.595+0000 7f2e8098f640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e6006b810 0x7f2e6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:28.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.595+0000 7f2e8098f640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e6006b810 0x7f2e6006dcd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c007fd0 tx=0x7f2e6c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:28.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.599+0000 7f2e71ffb640 1 -- 172.21.15.161:0/2857788601 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2e78079d90 con 0x7f2e7c06eca0 2024-03-23T14:35:28.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.811+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2e7c04f550 con 0x7f2e7c06eca0 2024-03-23T14:35:28.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.812+0000 7f2e71ffb640 1 -- 172.21.15.161:0/2857788601 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+73 (secure 0 0 0) 0x7f2e7807d450 con 0x7f2e7c06eca0 2024-03-23T14:35:28.808 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e6006b810 msgr2=0x7f2e6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e6006b810 0x7f2e6006dcd0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f2e6c007fd0 tx=0x7f2e6c009040 comp rx=0 tx=0).stop 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e7c06eca0 msgr2=0x7f2e7c1552e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e7c06eca0 0x7f2e7c1552e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f2e78000f80 tx=0x7f2e78004a40 comp rx=0 tx=0).stop 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 shutdown_connections 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e6006b810 0x7f2e6006dcd0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2e7c0724e0 0x7f2e7c1e5030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2e7c072100 0x7f2e7c1e4af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 --2- 172.21.15.161:0/2857788601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2e7c06eca0 0x7f2e7c1552e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 >> 172.21.15.161:0/2857788601 conn(0x7f2e7c0b17f0 msgr2=0x7f2e7c15c700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 shutdown_connections 2024-03-23T14:35:28.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:28.814+0000 7f2e833f4640 1 -- 172.21.15.161:0/2857788601 wait complete. 2024-03-23T14:35:28.820 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:35:28.820 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:35:28.820 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi183.front.sepia.ceph.com 2024-03-23T14:35:28.820 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-23T14:35:28.820 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:28.820 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:35:28.821 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:35:28.821 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:35:28.821 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-23T14:35:28.821 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:28.821 DEBUG:teuthology.orchestra.run.smithi183:> ip addr 2024-03-23T14:35:28.837 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:35:28.837 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:aa brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f0 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:ab brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f1 2024-03-23T14:35:28.838 INFO:teuthology.orchestra.run.smithi183.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:28.839 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:84 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:28.839 INFO:teuthology.orchestra.run.smithi183.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:35:28.839 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:85 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:28.839 INFO:teuthology.orchestra.run.smithi183.stdout: inet 172.21.15.183/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:35:28.839 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft 42733sec preferred_lft 42733sec 2024-03-23T14:35:28.839 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::ec4:7aff:fe88:8085/64 scope link 2024-03-23T14:35:28.839 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:28.839 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-23T14:35:28.840 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:28.840 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:28.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link add name ceph-brx type bridge 2024-03-23T14:35:28.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip addr flush dev ceph-brx 2024-03-23T14:35:28.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set ceph-brx up 2024-03-23T14:35:28.840 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-23T14:35:28.840 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:28.926 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:28 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:29.132 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:29.138 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:29.138 DEBUG:teuthology.orchestra.run.smithi183:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:35:29.169 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-23T14:35:29.170 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:29.170 DEBUG:teuthology.orchestra.run.smithi183:> route 2024-03-23T14:35:29.243 INFO:teuthology.orchestra.run.smithi183.stdout:Kernel IP routing table 2024-03-23T14:35:29.243 INFO:teuthology.orchestra.run.smithi183.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:35:29.243 INFO:teuthology.orchestra.run.smithi183.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:35:29.243 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:35:29.244 INFO:teuthology.orchestra.run.smithi183.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:35:29.244 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:29.244 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:29.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:35:29.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:35:29.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:35:29.244 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:29.331 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:29.536 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:29.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:29.544 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-03-23T14:35:29.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:29.561 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list-id 2024-03-23T14:35:29.615 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:29.616 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:29.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:29.616 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-23T14:35:29.616 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:29.708 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:29.737 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:29.743 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-23T14:35:29.744 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:29.744 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:29.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-23T14:35:29.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-23T14:35:29.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-23T14:35:29.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-23T14:35:29.744 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-23T14:35:29.744 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:29.802 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:29.904 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:29.910 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:29.911 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:29.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set brx.0 up 2024-03-23T14:35:29.911 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set dev brx.0 master ceph-brx 2024-03-23T14:35:29.911 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:29.958 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:29.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:29 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:29.991 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2024-03-23T14:35:29.992 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:35:29.992 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:30.007 INFO:teuthology.orchestra.run.smithi183.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-23T14:35:30.007 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:35:30.007 DEBUG:teuthology.orchestra.run.smithi183:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:30.061 DEBUG:teuthology.orchestra.run.smithi183:> sudo modprobe fuse 2024-03-23T14:35:30.128 DEBUG:teuthology.orchestra.run.smithi183:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:35:30.183 INFO:teuthology.orchestra.run.smithi183.stdout:/proc 2024-03-23T14:35:30.184 INFO:teuthology.orchestra.run.smithi183.stdout:/sys 2024-03-23T14:35:30.184 INFO:teuthology.orchestra.run.smithi183.stdout:/dev 2024-03-23T14:35:30.184 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/security 2024-03-23T14:35:30.184 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/shm 2024-03-23T14:35:30.184 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/pts 2024-03-23T14:35:30.184 INFO:teuthology.orchestra.run.smithi183.stdout:/run 2024-03-23T14:35:30.184 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/cgroup 2024-03-23T14:35:30.184 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/pstore 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/bpf 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/ 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/selinux 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/debug 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/mqueue 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/hugepages 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/tracing 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/fuse/connections 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:35:30.185 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/config 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/run/user/1000 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-4 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-5 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-6 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-7 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:30.186 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:30.187 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:30.187 DEBUG:teuthology.orchestra.run.smithi183:> ls /sys/fs/fuse/connections 2024-03-23T14:35:30.239 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T14:35:30.239 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.1 --id 1) 2024-03-23T14:35:30.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo modprobe fuse 2024-03-23T14:35:30.308 DEBUG:teuthology.orchestra.run.smithi183:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:35:30.364 INFO:teuthology.orchestra.run.smithi183.stdout:/proc 2024-03-23T14:35:30.364 INFO:teuthology.orchestra.run.smithi183.stdout:/sys 2024-03-23T14:35:30.364 INFO:teuthology.orchestra.run.smithi183.stdout:/dev 2024-03-23T14:35:30.364 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/security 2024-03-23T14:35:30.364 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/shm 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/pts 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/run 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/cgroup 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/pstore 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/bpf 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/ 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/selinux 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/debug 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/mqueue 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/hugepages 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/tracing 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/fuse/connections 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:35:30.365 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/config 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/run/user/1000 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-4 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-5 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-6 2024-03-23T14:35:30.366 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-7 2024-03-23T14:35:30.367 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns 2024-03-23T14:35:30.367 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:30.367 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:30.367 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:30.368 DEBUG:teuthology.orchestra.run.smithi183:> ls /sys/fs/fuse/connections 2024-03-23T14:35:30.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi183.stderr:2024-03-23T14:35:30.374+0000 7f8ced638480 -1 init, newargv = 0x55c6fe417810 newargc=15 2024-03-23T14:35:30.369 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi183.stderr:ceph-fuse[44192]: starting ceph client 2024-03-23T14:35:30.375 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi183.stderr:ceph-fuse[44192]: starting fuse 2024-03-23T14:35:30.384 INFO:teuthology.orchestra.run.smithi183.stdout:41 2024-03-23T14:35:30.384 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-23T14:35:30.385 DEBUG:teuthology.orchestra.run.smithi183:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:35:30.385 DEBUG:teuthology.orchestra.run.smithi183:> import glob 2024-03-23T14:35:30.385 DEBUG:teuthology.orchestra.run.smithi183:> import re 2024-03-23T14:35:30.385 DEBUG:teuthology.orchestra.run.smithi183:> import os 2024-03-23T14:35:30.385 DEBUG:teuthology.orchestra.run.smithi183:> import subprocess 2024-03-23T14:35:30.385 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-23T14:35:30.385 DEBUG:teuthology.orchestra.run.smithi183:> def _find_admin_socket(client_name): 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> files = glob.glob(asok_path) 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> mountpoint = "/home/ubuntu/cephtest/mnt.1" 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> # Given a non-glob path, it better be there 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> if "*" not in asok_path: 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> assert(len(files) == 1) 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> return files[0] 2024-03-23T14:35:30.386 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> for f in files: 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> contents = proc_f.read() 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> if mountpoint in contents: 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> return f 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> print(_find_admin_socket("client.1")) 2024-03-23T14:35:30.387 DEBUG:teuthology.orchestra.run.smithi183:> ' 2024-03-23T14:35:30.505 INFO:teuthology.orchestra.run.smithi183.stdout:/var/run/ceph/ceph-client.1.44192.asok 2024-03-23T14:35:30.508 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:30 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:30.516 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.44192.asok 2024-03-23T14:35:30.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:30.516 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.44192.asok status 2024-03-23T14:35:30.652 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-23T14:35:30.652 INFO:teuthology.orchestra.run.smithi183.stdout: "metadata": { 2024-03-23T14:35:30.652 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:35:30.652 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:35:30.652 INFO:teuthology.orchestra.run.smithi183.stdout: "entity_id": "1", 2024-03-23T14:35:30.652 INFO:teuthology.orchestra.run.smithi183.stdout: "hostname": "smithi183", 2024-03-23T14:35:30.652 INFO:teuthology.orchestra.run.smithi183.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2024-03-23T14:35:30.652 INFO:teuthology.orchestra.run.smithi183.stdout: "pid": "44192", 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: "root": "/" 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: "dentry_count": 0, 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: "dentry_pinned_count": 0, 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: "id": 4544, 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: "inst": { 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: "name": { 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "client", 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: "num": 4544 2024-03-23T14:35:30.653 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": { 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 952784726 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": { 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 952784726 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "inst_str": "client.4544 192.168.0.1:0/952784726", 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "addr_str": "192.168.0.1:0/952784726", 2024-03-23T14:35:30.654 INFO:teuthology.orchestra.run.smithi183.stdout: "inode_count": 1, 2024-03-23T14:35:30.655 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_epoch": 6, 2024-03-23T14:35:30.655 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_epoch": 20, 2024-03-23T14:35:30.655 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:35:30.655 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklisted": false, 2024-03-23T14:35:30.655 INFO:teuthology.orchestra.run.smithi183.stdout: "fs_name": "cephfs" 2024-03-23T14:35:30.655 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-23T14:35:30.660 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:35:30.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.777+0000 7f2abd610640 1 Processor -- start 2024-03-23T14:35:30.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.777+0000 7f2abd610640 1 -- start start 2024-03-23T14:35:30.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2abd610640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2ab80d1470 0x7f2ab80d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:30.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2abd610640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab80d2970 0x7f2ab80b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:30.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2abd610640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ab80d0460 con 0x7f2ab815e400 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2abd610640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f2ab80d1050 con 0x7f2ab80d1470 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2abd610640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab80d2e00 con 0x7f2ab80d2970 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2ab815e400 0x7f2ab80cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47866/0 (socket says 172.21.15.161:47866) 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab6ffd640 1 -- 172.21.15.161:0/3463511807 learned_addr learned my addr 172.21.15.161:0/3463511807 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab67fc640 1 --2- 172.21.15.161:0/3463511807 >> v2:172.21.15.183:3300/0 conn(0x7f2ab80d1470 0x7f2ab80d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab77fe640 1 --2- 172.21.15.161:0/3463511807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab80d2970 0x7f2ab80b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab5ffb640 1 -- 172.21.15.161:0/3463511807 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 717771447 0 0) 0x7f2ab80d0460 con 0x7f2ab815e400 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab5ffb640 1 -- 172.21.15.161:0/3463511807 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa0003520 con 0x7f2ab815e400 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab67fc640 1 -- 172.21.15.161:0/3463511807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab80d2970 msgr2=0x7f2ab80b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab67fc640 1 --2- 172.21.15.161:0/3463511807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab80d2970 0x7f2ab80b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab67fc640 1 -- 172.21.15.161:0/3463511807 >> v1:172.21.15.161:6789/0 conn(0x7f2ab815e400 legacy=0x7f2ab80cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:30.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.778+0000 7f2ab67fc640 1 -- 172.21.15.161:0/3463511807 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab80d06b0 con 0x7f2ab80d1470 2024-03-23T14:35:30.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.779+0000 7f2ab67fc640 1 --2- 172.21.15.161:0/3463511807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 0x7f2ab80d2430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0097a0 tx=0x7f2aac02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68d6a4eba471e25b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.779+0000 7f2ab5ffb640 1 -- 172.21.15.161:0/3463511807 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac031830 con 0x7f2ab80d1470 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.779+0000 7f2ab5ffb640 1 -- 172.21.15.161:0/3463511807 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac031990 con 0x7f2ab80d1470 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.779+0000 7f2ab5ffb640 1 -- 172.21.15.161:0/3463511807 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac031d20 con 0x7f2ab80d1470 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.780+0000 7f2abd610640 1 -- 172.21.15.161:0/3463511807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 msgr2=0x7f2ab80d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.780+0000 7f2abd610640 1 --2- 172.21.15.161:0/3463511807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 0x7f2ab80d2430 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0097a0 tx=0x7f2aac02b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.780+0000 7f2abd610640 1 -- 172.21.15.161:0/3463511807 shutdown_connections 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.780+0000 7f2abd610640 1 --2- 172.21.15.161:0/3463511807 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab80d2970 0x7f2ab80b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.780+0000 7f2abd610640 1 --2- 172.21.15.161:0/3463511807 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 0x7f2ab80d2430 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.780+0000 7f2abd610640 1 -- 172.21.15.161:0/3463511807 >> 172.21.15.161:0/3463511807 conn(0x7f2ab809d190 msgr2=0x7f2ab80d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.780+0000 7f2abd610640 1 -- 172.21.15.161:0/3463511807 shutdown_connections 2024-03-23T14:35:30.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.780+0000 7f2abd610640 1 -- 172.21.15.161:0/3463511807 wait complete. 2024-03-23T14:35:30.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.781+0000 7f2abd610640 1 Processor -- start 2024-03-23T14:35:30.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.781+0000 7f2abd610640 1 -- start start 2024-03-23T14:35:30.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2abd610640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 0x7f2ab815e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:30.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2abd610640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab80d2970 0x7f2ab8158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:30.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2abd610640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab815e400 0x7f2ab80b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:30.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2abd610640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab815ed20 con 0x7f2ab80d2970 2024-03-23T14:35:30.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2abd610640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab80b8700 con 0x7f2ab80d1470 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2abd610640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2ab8158ec0 con 0x7f2ab815e400 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2ab67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab80d2970 0x7f2ab8158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2ab67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab80d2970 0x7f2ab8158980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49526/0 (socket says 172.21.15.161:49526) 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2ab67fc640 1 -- 172.21.15.161:0/924383391 learned_addr learned my addr 172.21.15.161:0/924383391 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2ab6ffd640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 0x7f2ab815e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.782+0000 7f2ab77fe640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab815e400 0x7f2ab80b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2ab67fc640 1 -- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab815e400 msgr2=0x7f2ab80b6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2ab67fc640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab815e400 0x7f2ab80b6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2ab67fc640 1 -- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 msgr2=0x7f2ab815e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2ab67fc640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 0x7f2ab815e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2ab67fc640 1 -- 172.21.15.161:0/924383391 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aac009400 con 0x7f2ab80d2970 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2ab67fc640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab80d2970 0x7f2ab8158980 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0098c0 tx=0x7f2aac0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2a9f7fe640 1 -- 172.21.15.161:0/924383391 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac03a8f0 con 0x7f2ab80d2970 2024-03-23T14:35:30.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2a9f7fe640 1 -- 172.21.15.161:0/924383391 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2aac03aa70 con 0x7f2ab80d2970 2024-03-23T14:35:30.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab80b73e0 con 0x7f2ab80d2970 2024-03-23T14:35:30.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2a9f7fe640 1 -- 172.21.15.161:0/924383391 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aac03ae20 con 0x7f2ab80d2970 2024-03-23T14:35:30.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.783+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab80b79a0 con 0x7f2ab80d2970 2024-03-23T14:35:30.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.784+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab80a1e00 con 0x7f2ab80d2970 2024-03-23T14:35:30.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.784+0000 7f2a9f7fe640 1 -- 172.21.15.161:0/924383391 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2aac0423f0 con 0x7f2ab80d2970 2024-03-23T14:35:30.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.785+0000 7f2a9f7fe640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2aa406b810 0x7f2aa406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:30.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.785+0000 7f2a9f7fe640 1 -- 172.21.15.161:0/924383391 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f2aac0abe60 con 0x7f2ab80d2970 2024-03-23T14:35:30.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.785+0000 7f2ab6ffd640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2aa406b810 0x7f2aa406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:30.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.785+0000 7f2ab6ffd640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2aa406b810 0x7f2aa406dcd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8005880 tx=0x7f2aa8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:30.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.789+0000 7f2a9f7fe640 1 -- 172.21.15.161:0/924383391 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2aac076ae0 con 0x7f2ab80d2970 2024-03-23T14:35:30.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.996+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f2ab80b9c60 con 0x7f2ab80d2970 2024-03-23T14:35:30.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.996+0000 7f2a9f7fe640 1 -- 172.21.15.161:0/924383391 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v6) ==== 53+0+73 (secure 0 0 0) 0x7f2aac035020 con 0x7f2ab80d2970 2024-03-23T14:35:30.992 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T14:35:30.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2aa406b810 msgr2=0x7f2aa406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:30.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2aa406b810 0x7f2aa406dcd0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f2aa8005880 tx=0x7f2aa8009040 comp rx=0 tx=0).stop 2024-03-23T14:35:30.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab80d2970 msgr2=0x7f2ab8158980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:30.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab80d2970 0x7f2ab8158980 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f2aac0098c0 tx=0x7f2aac0046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 shutdown_connections 2024-03-23T14:35:30.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2aa406b810 0x7f2aa406dcd0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ab815e400 0x7f2ab80b6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ab80d2970 0x7f2ab8158980 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 --2- 172.21.15.161:0/924383391 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ab80d1470 0x7f2ab815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:30.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 >> 172.21.15.161:0/924383391 conn(0x7f2ab809d190 msgr2=0x7f2ab80b0ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:30.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 shutdown_connections 2024-03-23T14:35:30.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:30.998+0000 7f2abd610640 1 -- 172.21.15.161:0/924383391 wait complete. 2024-03-23T14:35:31.004 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:35:31.005 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:35:31.005 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi183.front.sepia.ceph.com 2024-03-23T14:35:31.005 INFO:tasks.cephfs.mount:self.client.name = client.3 2024-03-23T14:35:31.005 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.3 2024-03-23T14:35:31.005 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:35:31.005 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:35:31.005 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:35:31.005 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.3' 2024-03-23T14:35:31.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:31.005 DEBUG:teuthology.orchestra.run.smithi183:> ip addr 2024-03-23T14:35:31.021 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:35:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:35:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:35:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-03-23T14:35:31.022 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:aa brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f0 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:d9:38:ab brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: altname enp7s0f1 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:84 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 0c:c4:7a:88:80:85 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: inet 172.21.15.183/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft 42731sec preferred_lft 42731sec 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::ec4:7aff:fe88:8085/64 scope link 2024-03-23T14:35:31.023 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 02:d2:64:3b:8f:7e brd ff:ff:ff:ff:ff:ff 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::98f9:1fff:fe0c:b81c/64 scope link 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 02:d2:64:3b:8f:7e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::d2:64ff:fe3b:8f7e/64 scope link tentative 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:31.025 DEBUG:teuthology.orchestra.run.smithi183:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:35:31.087 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-23T14:35:31.089 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:31.089 DEBUG:teuthology.orchestra.run.smithi183:> route 2024-03-23T14:35:31.144 INFO:teuthology.orchestra.run.smithi183.stdout:Kernel IP routing table 2024-03-23T14:35:31.144 INFO:teuthology.orchestra.run.smithi183.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:35:31.144 INFO:teuthology.orchestra.run.smithi183.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:35:31.144 INFO:teuthology.orchestra.run.smithi183.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:35:31.144 INFO:teuthology.orchestra.run.smithi183.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:35:31.144 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:31.145 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:31.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:35:31.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:35:31.145 DEBUG:teuthology.orchestra.run.smithi183:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:35:31.145 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:31.232 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:31.273 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:31.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:31.280 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-03-23T14:35:31.295 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T14:35:31.295 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:31.295 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list-id 2024-03-23T14:35:31.350 INFO:teuthology.orchestra.run.smithi183.stdout:nsid 0 (iproute2 netns name: ceph-ns--home-ubuntu-cephtest-mnt.1) 2024-03-23T14:35:31.350 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:31.350 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:31.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T14:35:31.350 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.3 1 2024-03-23T14:35:31.350 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:31.439 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:31.466 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:31.472 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:31.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 92:f3:4c:8f:24:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.500 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::90f3:4cff:fe8f:24be/64 scope link tentative 2024-03-23T14:35:31.501 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.501 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:31.502 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 ::1/128 scope host 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout:2: veth0@if7: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout: link/ether 92:f3:4c:8f:24:be brd ff:ff:ff:ff:ff:ff link-netnsid 0 2024-03-23T14:35:31.571 INFO:teuthology.orchestra.run.smithi183.stdout: inet 192.168.0.1/16 brd 192.168.255.255 scope global veth0 2024-03-23T14:35:31.572 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.572 INFO:teuthology.orchestra.run.smithi183.stdout: inet6 fe80::90f3:4cff:fe8f:24be/64 scope link tentative 2024-03-23T14:35:31.572 INFO:teuthology.orchestra.run.smithi183.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:35:31.573 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.3' with 192.168.0.2/16 2024-03-23T14:35:31.573 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:31.573 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:31.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.3 type veth peer name brx.1 2024-03-23T14:35:31.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip addr add 192.168.0.2/16 brd 192.168.255.255 dev veth0 2024-03-23T14:35:31.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set veth0 up 2024-03-23T14:35:31.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip link set lo up 2024-03-23T14:35:31.573 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.3 ip route add default via 192.168.255.254 2024-03-23T14:35:31.573 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:31.662 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:31.733 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:31.740 DEBUG:teuthology.orchestra.run.smithi183:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:35:31.740 DEBUG:teuthology.orchestra.run.smithi183:> set -e 2024-03-23T14:35:31.740 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set brx.1 up 2024-03-23T14:35:31.740 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link set dev brx.1 master ceph-brx 2024-03-23T14:35:31.740 DEBUG:teuthology.orchestra.run.smithi183:> ') 2024-03-23T14:35:31.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:35:31.817 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:31.823 INFO:tasks.cephfs.fuse_mount:Client client.3 config is {} 2024-03-23T14:35:31.823 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:35:31.823 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -v /home/ubuntu/cephtest/mnt.3 2024-03-23T14:35:31.837 INFO:teuthology.orchestra.run.smithi183.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.3' 2024-03-23T14:35:31.837 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:35:31.838 DEBUG:teuthology.orchestra.run.smithi183:> chmod 0000 /home/ubuntu/cephtest/mnt.3 2024-03-23T14:35:31.892 DEBUG:teuthology.orchestra.run.smithi183:> sudo modprobe fuse 2024-03-23T14:35:31.963 DEBUG:teuthology.orchestra.run.smithi183:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:35:32.019 INFO:teuthology.orchestra.run.smithi183.stdout:/proc 2024-03-23T14:35:32.019 INFO:teuthology.orchestra.run.smithi183.stdout:/sys 2024-03-23T14:35:32.019 INFO:teuthology.orchestra.run.smithi183.stdout:/dev 2024-03-23T14:35:32.019 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/security 2024-03-23T14:35:32.019 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/shm 2024-03-23T14:35:32.019 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/pts 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/run 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/cgroup 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/pstore 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/bpf 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/ 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/selinux 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/debug 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/mqueue 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/hugepages 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/tracing 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/fuse/connections 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:35:32.020 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/config 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/run/user/1000 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-4 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-5 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-6 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-7 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:32.021 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T14:35:32.022 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T14:35:32.022 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.022 DEBUG:teuthology.orchestra.run.smithi183:> ls /sys/fs/fuse/connections 2024-03-23T14:35:32.076 INFO:teuthology.orchestra.run.smithi183.stdout:41 2024-03-23T14:35:32.077 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [41] 2024-03-23T14:35:32.077 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.3 --id 3) 2024-03-23T14:35:32.119 DEBUG:teuthology.orchestra.run.smithi183:> sudo modprobe fuse 2024-03-23T14:35:32.145 DEBUG:teuthology.orchestra.run.smithi183:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:35:32.200 INFO:teuthology.orchestra.run.smithi183.stdout:/proc 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/sys 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/dev 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/security 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/shm 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/pts 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/run 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/cgroup 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/pstore 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/bpf 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/ 2024-03-23T14:35:32.201 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/selinux 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/debug 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/mqueue 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/dev/hugepages 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/tracing 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/fs/fuse/connections 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/sys/kernel/config 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:35:32.202 INFO:teuthology.orchestra.run.smithi183.stdout:/run/user/1000 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-4 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-5 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-6 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/var/lib/ceph/osd/ceph-7 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run.smithi183.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T14:35:32.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.204 DEBUG:teuthology.orchestra.run.smithi183:> ls /sys/fs/fuse/connections 2024-03-23T14:35:32.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi183.stderr:2024-03-23T14:35:32.208+0000 7fcfe404c480 -1 init, newargv = 0x5616c3653810 newargc=15 2024-03-23T14:35:32.204 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi183.stderr:ceph-fuse[44648]: starting ceph client 2024-03-23T14:35:32.210 INFO:tasks.cephfs.fuse_mount.ceph-fuse.3.smithi183.stderr:ceph-fuse[44648]: starting fuse 2024-03-23T14:35:32.219 INFO:teuthology.orchestra.run.smithi183.stdout:41 2024-03-23T14:35:32.219 INFO:teuthology.orchestra.run.smithi183.stdout:44 2024-03-23T14:35:32.219 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41, 44] 2024-03-23T14:35:32.219 DEBUG:teuthology.orchestra.run.smithi183:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:35:32.219 DEBUG:teuthology.orchestra.run.smithi183:> import glob 2024-03-23T14:35:32.219 DEBUG:teuthology.orchestra.run.smithi183:> import re 2024-03-23T14:35:32.219 DEBUG:teuthology.orchestra.run.smithi183:> import os 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> import subprocess 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> def _find_admin_socket(client_name): 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> asok_path = "/var/run/ceph/ceph-client.3.*.asok" 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> files = glob.glob(asok_path) 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> mountpoint = "/home/ubuntu/cephtest/mnt.3" 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> # Given a non-glob path, it better be there 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> if "*" not in asok_path: 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> assert(len(files) == 1) 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> return files[0] 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> for f in files: 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:35:32.220 DEBUG:teuthology.orchestra.run.smithi183:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:35:32.221 DEBUG:teuthology.orchestra.run.smithi183:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:35:32.221 DEBUG:teuthology.orchestra.run.smithi183:> contents = proc_f.read() 2024-03-23T14:35:32.221 DEBUG:teuthology.orchestra.run.smithi183:> if mountpoint in contents: 2024-03-23T14:35:32.221 DEBUG:teuthology.orchestra.run.smithi183:> return f 2024-03-23T14:35:32.221 DEBUG:teuthology.orchestra.run.smithi183:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:35:32.221 DEBUG:teuthology.orchestra.run.smithi183:> 2024-03-23T14:35:32.221 DEBUG:teuthology.orchestra.run.smithi183:> print(_find_admin_socket("client.3")) 2024-03-23T14:35:32.221 DEBUG:teuthology.orchestra.run.smithi183:> ' 2024-03-23T14:35:32.340 INFO:teuthology.orchestra.run.smithi183.stdout:/var/run/ceph/ceph-client.3.44648.asok 2024-03-23T14:35:32.343 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-23T14:35:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:35:32.350 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.3.44648.asok 2024-03-23T14:35:32.351 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.3.44648.asok status 2024-03-23T14:35:32.487 INFO:teuthology.orchestra.run.smithi183.stdout:{ 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "metadata": { 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "entity_id": "3", 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "hostname": "smithi183", 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.3", 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "pid": "44648", 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "root": "/" 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "dentry_count": 0, 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "dentry_pinned_count": 0, 2024-03-23T14:35:32.488 INFO:teuthology.orchestra.run.smithi183.stdout: "id": 4660, 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "inst": { 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "name": { 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "client", 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "num": 4660 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": { 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "192.168.0.2:0", 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 905626399 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: } 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": { 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "type": "v1", 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "addr": "192.168.0.2:0", 2024-03-23T14:35:32.489 INFO:teuthology.orchestra.run.smithi183.stdout: "nonce": 905626399 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: }, 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: "inst_str": "client.4660 192.168.0.2:0/905626399", 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: "addr_str": "192.168.0.2:0/905626399", 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: "inode_count": 1, 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: "mds_epoch": 6, 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_epoch": 20, 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: "blocklisted": false, 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout: "fs_name": "cephfs" 2024-03-23T14:35:32.490 INFO:teuthology.orchestra.run.smithi183.stdout:} 2024-03-23T14:35:32.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.496 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-23T14:35:32.496 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:32.515 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-23T14:35:32.516 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:32.516 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.516 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:32.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.589 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-23T14:35:32.589 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.2 2024-03-23T14:35:32.646 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-23T14:35:32.646 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2024-03-23T14:35:32.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.646 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.2 2024-03-23T14:35:32.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.718 DEBUG:teuthology.orchestra.run.smithi183:> stat --file-system '--printf=%T 2024-03-23T14:35:32.718 DEBUG:teuthology.orchestra.run.smithi183:> ' -- /home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:32.733 INFO:teuthology.orchestra.run.smithi183.stdout:fuseblk 2024-03-23T14:35:32.734 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:32.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.734 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:32.800 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.800 DEBUG:teuthology.orchestra.run.smithi183:> stat --file-system '--printf=%T 2024-03-23T14:35:32.800 DEBUG:teuthology.orchestra.run.smithi183:> ' -- /home/ubuntu/cephtest/mnt.3 2024-03-23T14:35:32.855 INFO:teuthology.orchestra.run.smithi183.stdout:fuseblk 2024-03-23T14:35:32.856 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.3 2024-03-23T14:35:32.856 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:32.856 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.3 2024-03-23T14:35:32.922 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2024-03-23T14:35:32.941 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:33.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.054+0000 7f3ae275e640 1 Processor -- start 2024-03-23T14:35:33.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.054+0000 7f3ae275e640 1 -- start start 2024-03-23T14:35:33.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3ae275e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3adc1601f0 0x7f3adc072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3ae275e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3adc1607d0 0x7f3adc15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3ae275e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3adc0cf4f0 con 0x7f3adc145080 2024-03-23T14:35:33.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3ae275e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3adc071010 con 0x7f3adc1607d0 2024-03-23T14:35:33.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3ae275e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f3adc0cef30 con 0x7f3adc1601f0 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3ae0cfb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3adc1607d0 0x7f3adc15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3adbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3adc145080 0x7f3adc15fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47874/0 (socket says 172.21.15.161:47874) 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3adbfff640 1 -- 172.21.15.161:0/2521426944 learned_addr learned my addr 172.21.15.161:0/2521426944 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3adb7fe640 1 --2- 172.21.15.161:0/2521426944 >> v2:172.21.15.183:3300/0 conn(0x7f3adc1601f0 0x7f3adc072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3ae0cfb640 1 -- 172.21.15.161:0/2521426944 >> v2:172.21.15.183:3300/0 conn(0x7f3adc1601f0 msgr2=0x7f3adc072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.055+0000 7f3ae0cfb640 1 --2- 172.21.15.161:0/2521426944 >> v2:172.21.15.183:3300/0 conn(0x7f3adc1601f0 0x7f3adc072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.056+0000 7f3ae0cfb640 1 -- 172.21.15.161:0/2521426944 >> v1:172.21.15.161:6789/0 conn(0x7f3adc145080 legacy=0x7f3adc15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.056+0000 7f3ae0cfb640 1 -- 172.21.15.161:0/2521426944 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3adc0cf210 con 0x7f3adc1607d0 2024-03-23T14:35:33.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.056+0000 7f3adaffd640 1 -- 172.21.15.161:0/2521426944 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 76516869 0 0) 0x7f3adc0cf4f0 con 0x7f3adc145080 2024-03-23T14:35:33.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.056+0000 7f3ae0cfb640 1 --2- 172.21.15.161:0/2521426944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1607d0 0x7f3adc15c4f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3adc073910 tx=0x7f3ad402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a94d8be51f035a0c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.056+0000 7f3adaffd640 1 -- 172.21.15.161:0/2521426944 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad402cbf0 con 0x7f3adc1607d0 2024-03-23T14:35:33.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.056+0000 7f3adaffd640 1 -- 172.21.15.161:0/2521426944 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ad402cd50 con 0x7f3adc1607d0 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.056+0000 7f3adaffd640 1 -- 172.21.15.161:0/2521426944 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad4034660 con 0x7f3adc1607d0 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.057+0000 7f3ae275e640 1 -- 172.21.15.161:0/2521426944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1607d0 msgr2=0x7f3adc15c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.057+0000 7f3ae275e640 1 --2- 172.21.15.161:0/2521426944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1607d0 0x7f3adc15c4f0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3adc073910 tx=0x7f3ad402be40 comp rx=0 tx=0).stop 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.057+0000 7f3ae275e640 1 -- 172.21.15.161:0/2521426944 shutdown_connections 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.057+0000 7f3ae275e640 1 --2- 172.21.15.161:0/2521426944 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1607d0 0x7f3adc15c4f0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.057+0000 7f3ae275e640 1 --2- 172.21.15.161:0/2521426944 >> v2:172.21.15.183:3300/0 conn(0x7f3adc1601f0 0x7f3adc072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.057+0000 7f3ae275e640 1 -- 172.21.15.161:0/2521426944 >> 172.21.15.161:0/2521426944 conn(0x7f3adc0b17f0 msgr2=0x7f3adc0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.057+0000 7f3ae275e640 1 -- 172.21.15.161:0/2521426944 shutdown_connections 2024-03-23T14:35:33.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.057+0000 7f3ae275e640 1 -- 172.21.15.161:0/2521426944 wait complete. 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.058+0000 7f3ae275e640 1 Processor -- start 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.058+0000 7f3ae275e640 1 -- start start 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3ae275e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3adc145080 0x7f3adc1f1880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3ae275e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1601f0 0x7f3adc140140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3ae275e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3adc1607d0 0x7f3adc1f2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3ae275e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc13aed0 con 0x7f3adc1601f0 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3ae275e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3adc140680 con 0x7f3adc145080 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3ae275e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3adc1f2890 con 0x7f3adc1607d0 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3adb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1601f0 0x7f3adc140140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3adb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1601f0 0x7f3adc140140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49540/0 (socket says 172.21.15.161:49540) 2024-03-23T14:35:33.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3adb7fe640 1 -- 172.21.15.161:0/1935199221 learned_addr learned my addr 172.21.15.161:0/1935199221 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3adbfff640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3adc145080 0x7f3adc1f1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.059+0000 7f3ae0cfb640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3adc1607d0 0x7f3adc1f2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3adb7fe640 1 -- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3adc1607d0 msgr2=0x7f3adc1f2350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3adb7fe640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3adc1607d0 0x7f3adc1f2350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3adb7fe640 1 -- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3adc145080 msgr2=0x7f3adc1f1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3adb7fe640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3adc145080 0x7f3adc1f1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3adb7fe640 1 -- 172.21.15.161:0/1935199221 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ad4009a70 con 0x7f3adc1601f0 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3adbfff640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3adc145080 0x7f3adc1f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3adb7fe640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1601f0 0x7f3adc140140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000e9e0 tx=0x7f3ad000eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3ad8ff9640 1 -- 172.21.15.161:0/1935199221 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad0009670 con 0x7f3adc1601f0 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3ad8ff9640 1 -- 172.21.15.161:0/1935199221 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ad000c9d0 con 0x7f3adc1601f0 2024-03-23T14:35:33.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3ad8ff9640 1 -- 172.21.15.161:0/1935199221 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ad000cd80 con 0x7f3adc1601f0 2024-03-23T14:35:33.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3adc140900 con 0x7f3adc1601f0 2024-03-23T14:35:33.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.060+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3adc140ec0 con 0x7f3adc1601f0 2024-03-23T14:35:33.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.061+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3adc13b080 con 0x7f3adc1601f0 2024-03-23T14:35:33.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.061+0000 7f3ad8ff9640 1 -- 172.21.15.161:0/1935199221 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ad0013470 con 0x7f3adc1601f0 2024-03-23T14:35:33.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.062+0000 7f3ad8ff9640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3ac806b810 0x7f3ac806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.062+0000 7f3ad8ff9640 1 -- 172.21.15.161:0/1935199221 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3ad008a3a0 con 0x7f3adc1601f0 2024-03-23T14:35:33.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.062+0000 7f3adbfff640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3ac806b810 0x7f3ac806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.062+0000 7f3adbfff640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3ac806b810 0x7f3ac806dcd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3acc000f60 tx=0x7f3acc007d50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.066+0000 7f3ad8ff9640 1 -- 172.21.15.161:0/1935199221 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3adc13b080 con 0x7f3adc1601f0 2024-03-23T14:35:33.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.237+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3adc13b080 con 0x7f3adc1601f0 2024-03-23T14:35:33.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.238+0000 7f3ad8ff9640 1 -- 172.21.15.161:0/1935199221 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f3adc13b080 con 0x7f3adc1601f0 2024-03-23T14:35:33.234 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:33.234 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:33.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3ac806b810 msgr2=0x7f3ac806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3ac806b810 0x7f3ac806dcd0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3acc000f60 tx=0x7f3acc007d50 comp rx=0 tx=0).stop 2024-03-23T14:35:33.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1601f0 msgr2=0x7f3adc140140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1601f0 0x7f3adc140140 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3ad000e9e0 tx=0x7f3ad000eea0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 shutdown_connections 2024-03-23T14:35:33.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3ac806b810 0x7f3ac806dcd0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3adc1607d0 0x7f3adc1f2350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3adc1601f0 0x7f3adc140140 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 --2- 172.21.15.161:0/1935199221 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3adc145080 0x7f3adc1f1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 >> 172.21.15.161:0/1935199221 conn(0x7f3adc0b17f0 msgr2=0x7f3adc13e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:33.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 shutdown_connections 2024-03-23T14:35:33.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.240+0000 7f3ae275e640 1 -- 172.21.15.161:0/1935199221 wait complete. 2024-03-23T14:35:33.247 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:33.248 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:33.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.356+0000 7f21d63e2640 1 Processor -- start 2024-03-23T14:35:33.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.356+0000 7f21d63e2640 1 -- start start 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.356+0000 7f21d63e2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f21d01552f0 0x7f21d00b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.356+0000 7f21d63e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00d0380 0x7f21d006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.356+0000 7f21d63e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21d015f480 con 0x7f21d00b2aa0 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.356+0000 7f21d63e2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f21d01556d0 con 0x7f21d01552f0 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.356+0000 7f21d63e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f21d015e870 con 0x7f21d00d0380 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21cffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f21d01552f0 0x7f21d00b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21d497f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00d0380 0x7f21d006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21cffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f21d01552f0 0x7f21d00b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:38560/0 (socket says 172.21.15.161:38560) 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21cffff640 1 -- 172.21.15.161:0/846949538 learned_addr learned my addr 172.21.15.161:0/846949538 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21cf7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f21d00b2aa0 0x7f21d00cfc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:60900/0 (socket says 172.21.15.161:60900) 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21cffff640 1 -- 172.21.15.161:0/846949538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00d0380 msgr2=0x7f21d006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:33.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21cffff640 1 --2- 172.21.15.161:0/846949538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00d0380 0x7f21d006ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21cffff640 1 -- 172.21.15.161:0/846949538 >> v1:172.21.15.183:6789/0 conn(0x7f21d00b2aa0 legacy=0x7f21d00cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.357+0000 7f21cffff640 1 -- 172.21.15.161:0/846949538 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21d015eb50 con 0x7f21d01552f0 2024-03-23T14:35:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21cffff640 1 --2- 172.21.15.161:0/846949538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d01552f0 0x7f21d00b3d10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f21c0009870 tx=0x7f21c002b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=163f100cb371a33b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21ceffd640 1 -- 172.21.15.161:0/846949538 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0030840 con 0x7f21d01552f0 2024-03-23T14:35:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21ceffd640 1 -- 172.21.15.161:0/846949538 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21c00309a0 con 0x7f21d01552f0 2024-03-23T14:35:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21ceffd640 1 -- 172.21.15.161:0/846949538 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c0030d30 con 0x7f21d01552f0 2024-03-23T14:35:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21d63e2640 1 -- 172.21.15.161:0/846949538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d01552f0 msgr2=0x7f21d00b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21d63e2640 1 --2- 172.21.15.161:0/846949538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d01552f0 0x7f21d00b3d10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f21c0009870 tx=0x7f21c002b980 comp rx=0 tx=0).stop 2024-03-23T14:35:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21d63e2640 1 -- 172.21.15.161:0/846949538 shutdown_connections 2024-03-23T14:35:33.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21d63e2640 1 --2- 172.21.15.161:0/846949538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00d0380 0x7f21d006ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21d63e2640 1 --2- 172.21.15.161:0/846949538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d01552f0 0x7f21d00b3d10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21d63e2640 1 -- 172.21.15.161:0/846949538 >> 172.21.15.161:0/846949538 conn(0x7f21d006b260 msgr2=0x7f21d006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:33.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21d63e2640 1 -- 172.21.15.161:0/846949538 shutdown_connections 2024-03-23T14:35:33.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.358+0000 7f21d63e2640 1 -- 172.21.15.161:0/846949538 wait complete. 2024-03-23T14:35:33.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.359+0000 7f21d63e2640 1 Processor -- start 2024-03-23T14:35:33.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.360+0000 7f21d63e2640 1 -- start start 2024-03-23T14:35:33.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.360+0000 7f21d63e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00b2aa0 0x7f21d0153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.360+0000 7f21d63e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00d0380 0x7f21d0154650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.360+0000 7f21d63e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d01552f0 0x7f21d015b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.360+0000 7f21d63e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f21d0146920 con 0x7f21d01552f0 2024-03-23T14:35:33.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.360+0000 7f21d63e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f21d0154b90 con 0x7f21d00d0380 2024-03-23T14:35:33.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.360+0000 7f21d63e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f21d0152cb0 con 0x7f21d00b2aa0 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d01552f0 0x7f21d015b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d01552f0 0x7f21d015b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49556/0 (socket says 172.21.15.161:49556) 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 -- 172.21.15.161:0/69178814 learned_addr learned my addr 172.21.15.161:0/69178814 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21cffff640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00b2aa0 0x7f21d0153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21cf7fe640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00d0380 0x7f21d0154650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 -- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00b2aa0 msgr2=0x7f21d0153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00b2aa0 0x7f21d0153a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 -- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00d0380 msgr2=0x7f21d0154650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00d0380 0x7f21d0154650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 -- 172.21.15.161:0/69178814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21c00094d0 con 0x7f21d01552f0 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21cf7fe640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00d0380 0x7f21d0154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d497f640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d01552f0 0x7f21d015b4a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f21c800efc0 tx=0x7f21c800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21ccff9640 1 -- 172.21.15.161:0/69178814 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8014480 con 0x7f21d01552f0 2024-03-23T14:35:33.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21ccff9640 1 -- 172.21.15.161:0/69178814 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21c800f040 con 0x7f21d01552f0 2024-03-23T14:35:33.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21d0152f50 con 0x7f21d01552f0 2024-03-23T14:35:33.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21ccff9640 1 -- 172.21.15.161:0/69178814 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21c8014820 con 0x7f21d01552f0 2024-03-23T14:35:33.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.361+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21d0153510 con 0x7f21d01552f0 2024-03-23T14:35:33.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.362+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21d0153730 con 0x7f21d01552f0 2024-03-23T14:35:33.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.363+0000 7f21ccff9640 1 -- 172.21.15.161:0/69178814 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f21c8014980 con 0x7f21d01552f0 2024-03-23T14:35:33.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.363+0000 7f21ccff9640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21b006b810 0x7f21b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.363+0000 7f21ccff9640 1 -- 172.21.15.161:0/69178814 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f21c808ac90 con 0x7f21d01552f0 2024-03-23T14:35:33.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.363+0000 7f21cffff640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21b006b810 0x7f21b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.363+0000 7f21cffff640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21b006b810 0x7f21b006dcd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f21c0036000 tx=0x7f21c0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.367+0000 7f21ccff9640 1 -- 172.21.15.161:0/69178814 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f21c8055760 con 0x7f21d01552f0 2024-03-23T14:35:33.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.538+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f21d004f4e0 con 0x7f21d01552f0 2024-03-23T14:35:33.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.539+0000 7f21ccff9640 1 -- 172.21.15.161:0/69178814 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f21c80180a0 con 0x7f21d01552f0 2024-03-23T14:35:33.534 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21b006b810 msgr2=0x7f21b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21b006b810 0x7f21b006dcd0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f21c0036000 tx=0x7f21c0008030 comp rx=0 tx=0).stop 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d01552f0 msgr2=0x7f21d015b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d01552f0 0x7f21d015b4a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f21c800efc0 tx=0x7f21c800c490 comp rx=0 tx=0).stop 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 shutdown_connections 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21b006b810 0x7f21b006dcd0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21d01552f0 0x7f21d015b4a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21d00d0380 0x7f21d0154650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 --2- 172.21.15.161:0/69178814 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21d00b2aa0 0x7f21d0153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 >> 172.21.15.161:0/69178814 conn(0x7f21d006b260 msgr2=0x7f21d00cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:33.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 shutdown_connections 2024-03-23T14:35:33.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.541+0000 7f21d63e2640 1 -- 172.21.15.161:0/69178814 wait complete. 2024-03-23T14:35:33.547 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:33.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.654+0000 7f8346593640 1 Processor -- start 2024-03-23T14:35:33.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.654+0000 7f8346593640 1 -- start start 2024-03-23T14:35:33.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f8346593640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83401549c0 0x7f834006e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f8346593640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f834015b780 0x7f834006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f8346593640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f83400b2c20 con 0x7f8340070c20 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f8346593640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f834006ee90 con 0x7f834015b780 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f8346593640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f83400b2a10 con 0x7f83401549c0 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f833f7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8340070c20 0x7f834015b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47880/0 (socket says 172.21.15.161:47880) 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f833f7fe640 1 -- 172.21.15.161:0/3218333400 learned_addr learned my addr 172.21.15.161:0/3218333400 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f833ffff640 1 --2- 172.21.15.161:0/3218333400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83401549c0 0x7f834006e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f8344b30640 1 --2- 172.21.15.161:0/3218333400 >> v2:172.21.15.183:3300/0 conn(0x7f834015b780 0x7f834006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f833effd640 1 -- 172.21.15.161:0/3218333400 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3943321555 0 0) 0x7f83400b2c20 con 0x7f8340070c20 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.655+0000 7f833effd640 1 -- 172.21.15.161:0/3218333400 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8328003520 con 0x7f8340070c20 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.656+0000 7f8344b30640 1 -- 172.21.15.161:0/3218333400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83401549c0 msgr2=0x7f834006e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.656+0000 7f8344b30640 1 --2- 172.21.15.161:0/3218333400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83401549c0 0x7f834006e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.656+0000 7f8344b30640 1 -- 172.21.15.161:0/3218333400 >> v1:172.21.15.161:6789/0 conn(0x7f8340070c20 legacy=0x7f834015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.656+0000 7f8344b30640 1 -- 172.21.15.161:0/3218333400 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83400b2f00 con 0x7f834015b780 2024-03-23T14:35:33.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.656+0000 7f8344b30640 1 --2- 172.21.15.161:0/3218333400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 0x7f834006d6f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8338009d90 tx=0x7f833802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57bd322ead5fef2f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.656+0000 7f833effd640 1 -- 172.21.15.161:0/3218333400 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f833802cc50 con 0x7f834015b780 2024-03-23T14:35:33.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.656+0000 7f833effd640 1 -- 172.21.15.161:0/3218333400 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f833802cdb0 con 0x7f834015b780 2024-03-23T14:35:33.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f833effd640 1 -- 172.21.15.161:0/3218333400 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8338034640 con 0x7f834015b780 2024-03-23T14:35:33.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f8346593640 1 -- 172.21.15.161:0/3218333400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 msgr2=0x7f834006d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f8346593640 1 --2- 172.21.15.161:0/3218333400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 0x7f834006d6f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f8338009d90 tx=0x7f833802bea0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f8346593640 1 -- 172.21.15.161:0/3218333400 shutdown_connections 2024-03-23T14:35:33.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f8346593640 1 --2- 172.21.15.161:0/3218333400 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 0x7f834006d6f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f8346593640 1 --2- 172.21.15.161:0/3218333400 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f83401549c0 0x7f834006e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f8346593640 1 -- 172.21.15.161:0/3218333400 >> 172.21.15.161:0/3218333400 conn(0x7f834006b4a0 msgr2=0x7f834006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:33.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f8346593640 1 -- 172.21.15.161:0/3218333400 shutdown_connections 2024-03-23T14:35:33.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.657+0000 7f8346593640 1 -- 172.21.15.161:0/3218333400 wait complete. 2024-03-23T14:35:33.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.658+0000 7f8346593640 1 Processor -- start 2024-03-23T14:35:33.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.658+0000 7f8346593640 1 -- start start 2024-03-23T14:35:33.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f8346593640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8340070c20 0x7f834014cb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f8346593640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83401549c0 0x7f83400b05f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f8346593640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 0x7f83400b0b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f8346593640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f834014db90 con 0x7f83401549c0 2024-03-23T14:35:33.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f8346593640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8340150b60 con 0x7f834015b780 2024-03-23T14:35:33.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f8346593640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8340150db0 con 0x7f8340070c20 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83401549c0 0x7f83400b05f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83401549c0 0x7f83400b05f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49572/0 (socket says 172.21.15.161:49572) 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833f7fe640 1 -- 172.21.15.161:0/749041220 learned_addr learned my addr 172.21.15.161:0/749041220 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833ffff640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8340070c20 0x7f834014cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f8344b30640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 0x7f83400b0b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833f7fe640 1 -- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8340070c20 msgr2=0x7f834014cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833f7fe640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8340070c20 0x7f834014cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833f7fe640 1 -- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 msgr2=0x7f83400b0b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833f7fe640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 0x7f83400b0b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.659+0000 7f833f7fe640 1 -- 172.21.15.161:0/749041220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8338009a70 con 0x7f83401549c0 2024-03-23T14:35:33.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.660+0000 7f833f7fe640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83401549c0 0x7f83400b05f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f833400c430 tx=0x7f833400c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.660+0000 7f833cff9640 1 -- 172.21.15.161:0/749041220 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8334005ce0 con 0x7f83401549c0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.660+0000 7f833cff9640 1 -- 172.21.15.161:0/749041220 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8334005e40 con 0x7f83401549c0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.660+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8340159000 con 0x7f83401549c0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.660+0000 7f833cff9640 1 -- 172.21.15.161:0/749041220 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83340072f0 con 0x7f83401549c0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.660+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83401595c0 con 0x7f83401549c0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.661+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8340073570 con 0x7f83401549c0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.661+0000 7f833cff9640 1 -- 172.21.15.161:0/749041220 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8334007450 con 0x7f83401549c0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.662+0000 7f833cff9640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f832c06b810 0x7f832c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.662+0000 7f833cff9640 1 -- 172.21.15.161:0/749041220 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8334089780 con 0x7f83401549c0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.662+0000 7f833ffff640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f832c06b810 0x7f832c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.662+0000 7f833ffff640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f832c06b810 0x7f832c06dcd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8330009b40 tx=0x7f8330009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.666+0000 7f833cff9640 1 -- 172.21.15.161:0/749041220 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f83340542e0 con 0x7f83401549c0 2024-03-23T14:35:33.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.837+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f834004f550 con 0x7f83401549c0 2024-03-23T14:35:33.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.837+0000 7f833cff9640 1 -- 172.21.15.161:0/749041220 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f83340579a0 con 0x7f83401549c0 2024-03-23T14:35:33.833 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:33.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f832c06b810 msgr2=0x7f832c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f832c06b810 0x7f832c06dcd0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8330009b40 tx=0x7f8330009190 comp rx=0 tx=0).stop 2024-03-23T14:35:33.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83401549c0 msgr2=0x7f83400b05f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83401549c0 0x7f83400b05f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f833400c430 tx=0x7f833400c8f0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 shutdown_connections 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f832c06b810 0x7f832c06dcd0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f834015b780 0x7f83400b0b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f83401549c0 0x7f83400b05f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 --2- 172.21.15.161:0/749041220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8340070c20 0x7f834014cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 >> 172.21.15.161:0/749041220 conn(0x7f834006b4a0 msgr2=0x7f834006b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 shutdown_connections 2024-03-23T14:35:33.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.839+0000 7f8346593640 1 -- 172.21.15.161:0/749041220 wait complete. 2024-03-23T14:35:33.846 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:33.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.953+0000 7ffa46fdd640 1 Processor -- start 2024-03-23T14:35:33.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.953+0000 7ffa46fdd640 1 -- start start 2024-03-23T14:35:33.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa46fdd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffa40161820 0x7ffa400d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa46fdd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffa400d2ba0 0x7ffa400b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa46fdd640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa40161c00 con 0x7ffa400d1270 2024-03-23T14:35:33.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa46fdd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffa400cea30 con 0x7ffa400d2ba0 2024-03-23T14:35:33.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa46fdd640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7ffa400cfd70 con 0x7ffa40161820 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa4557a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffa400d2ba0 0x7ffa400b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa4557a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffa400d2ba0 0x7ffa400b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49582/0 (socket says 172.21.15.161:49582) 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa4557a640 1 -- 172.21.15.161:0/2052449576 learned_addr learned my addr 172.21.15.161:0/2052449576 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.954+0000 7ffa44d79640 1 --2- 172.21.15.161:0/2052449576 >> v2:172.21.15.183:3300/0 conn(0x7ffa40161820 0x7ffa400d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.955+0000 7ffa4557a640 1 -- 172.21.15.161:0/2052449576 >> v2:172.21.15.183:3300/0 conn(0x7ffa40161820 msgr2=0x7ffa400d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.955+0000 7ffa4557a640 1 --2- 172.21.15.161:0/2052449576 >> v2:172.21.15.183:3300/0 conn(0x7ffa40161820 0x7ffa400d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.955+0000 7ffa4557a640 1 -- 172.21.15.161:0/2052449576 >> v1:172.21.15.183:6789/0 conn(0x7ffa400d1270 legacy=0x7ffa400d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.955+0000 7ffa4557a640 1 -- 172.21.15.161:0/2052449576 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa400cecd0 con 0x7ffa400d2ba0 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.955+0000 7ffa4557a640 1 --2- 172.21.15.161:0/2052449576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa400b0a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffa4006b270 tx=0x7ffa3802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b792ad351f1ae01 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.955+0000 7ffa3f7fe640 1 -- 172.21.15.161:0/2052449576 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa3802cb20 con 0x7ffa400d2ba0 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa3f7fe640 1 -- 172.21.15.161:0/2052449576 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa3802cc80 con 0x7ffa400d2ba0 2024-03-23T14:35:33.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa3f7fe640 1 -- 172.21.15.161:0/2052449576 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa380343f0 con 0x7ffa400d2ba0 2024-03-23T14:35:33.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa46fdd640 1 -- 172.21.15.161:0/2052449576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 msgr2=0x7ffa400b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/2052449576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa400b0a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7ffa4006b270 tx=0x7ffa3802bf80 comp rx=0 tx=0).stop 2024-03-23T14:35:33.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa46fdd640 1 -- 172.21.15.161:0/2052449576 shutdown_connections 2024-03-23T14:35:33.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/2052449576 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa400b0a80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/2052449576 >> v2:172.21.15.183:3300/0 conn(0x7ffa40161820 0x7ffa400d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa46fdd640 1 -- 172.21.15.161:0/2052449576 >> 172.21.15.161:0/2052449576 conn(0x7ffa4009d190 msgr2=0x7ffa400d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:33.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa46fdd640 1 -- 172.21.15.161:0/2052449576 shutdown_connections 2024-03-23T14:35:33.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.956+0000 7ffa46fdd640 1 -- 172.21.15.161:0/2052449576 wait complete. 2024-03-23T14:35:33.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.957+0000 7ffa46fdd640 1 Processor -- start 2024-03-23T14:35:33.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.957+0000 7ffa46fdd640 1 -- start start 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa46fdd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa400d1270 0x7ffa4015cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa46fdd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa4015c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa46fdd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa40161820 0x7ffa400c9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa46fdd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffa400c8c20 con 0x7ffa400d2ba0 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa46fdd640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffa400c7fa0 con 0x7ffa40161820 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa46fdd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ffa400c8280 con 0x7ffa400d1270 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa4015c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa3ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa4015c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49592/0 (socket says 172.21.15.161:49592) 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa3ffff640 1 -- 172.21.15.161:0/4071034948 learned_addr learned my addr 172.21.15.161:0/4071034948 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:33.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa44d79640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa400d1270 0x7ffa4015cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.958+0000 7ffa4557a640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa40161820 0x7ffa400c9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3ffff640 1 -- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa400d1270 msgr2=0x7ffa4015cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3ffff640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa400d1270 0x7ffa4015cf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3ffff640 1 -- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa40161820 msgr2=0x7ffa400c9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3ffff640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa40161820 0x7ffa400c9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3ffff640 1 -- 172.21.15.161:0/4071034948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa38009b30 con 0x7ffa400d2ba0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3ffff640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa4015c9b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400c440 tx=0x7ffa3400c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3d7fa640 1 -- 172.21.15.161:0/4071034948 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa340052d0 con 0x7ffa400d2ba0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3d7fa640 1 -- 172.21.15.161:0/4071034948 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa34005430 con 0x7ffa400d2ba0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa400c8560 con 0x7ffa400d2ba0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa3d7fa640 1 -- 172.21.15.161:0/4071034948 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa34007260 con 0x7ffa400d2ba0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.959+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa4015d680 con 0x7ffa400d2ba0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.960+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa400c8d60 con 0x7ffa400d2ba0 2024-03-23T14:35:33.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.960+0000 7ffa3d7fa640 1 -- 172.21.15.161:0/4071034948 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffa340073c0 con 0x7ffa400d2ba0 2024-03-23T14:35:33.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.961+0000 7ffa3d7fa640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:33.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.961+0000 7ffa3d7fa640 1 -- 172.21.15.161:0/4071034948 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ffa340893e0 con 0x7ffa400d2ba0 2024-03-23T14:35:33.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.961+0000 7ffa44d79640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:33.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.961+0000 7ffa44d79640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffa40068d70 tx=0x7ffa30009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:33.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:33.965+0000 7ffa3d7fa640 1 -- 172.21.15.161:0/4071034948 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffa400c8d60 con 0x7ffa400d2ba0 2024-03-23T14:35:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.139+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffa400c8d60 con 0x7ffa400d2ba0 2024-03-23T14:35:34.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.140+0000 7ffa3d7fa640 1 -- 172.21.15.161:0/4071034948 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7ffa400c8d60 con 0x7ffa400d2ba0 2024-03-23T14:35:34.136 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffa2c06b810 msgr2=0x7ffa2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ffa40068d70 tx=0x7ffa30009190 comp rx=0 tx=0).stop 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 msgr2=0x7ffa4015c9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa4015c9b0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ffa3400c440 tx=0x7ffa3400c900 comp rx=0 tx=0).stop 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 shutdown_connections 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffa2c06b810 0x7ffa2c06dcd0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffa40161820 0x7ffa400c9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffa400d2ba0 0x7ffa4015c9b0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 --2- 172.21.15.161:0/4071034948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffa400d1270 0x7ffa4015cf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 >> 172.21.15.161:0/4071034948 conn(0x7ffa4009d190 msgr2=0x7ffa4015e2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 shutdown_connections 2024-03-23T14:35:34.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.142+0000 7ffa46fdd640 1 -- 172.21.15.161:0/4071034948 wait complete. 2024-03-23T14:35:34.148 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:34.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.254+0000 7f462686d640 1 Processor -- start 2024-03-23T14:35:34.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.254+0000 7f462686d640 1 -- start start 2024-03-23T14:35:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f462686d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4620162970 0x7f462015ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f462686d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f46200d2a70 0x7f462015e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f462686d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46200d2270 con 0x7f46200d2690 2024-03-23T14:35:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f462686d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f46200d06a0 con 0x7f46200d2a70 2024-03-23T14:35:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f462686d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f46200d0980 con 0x7f4620162970 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f461ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f46200d2690 0x7f46200d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47884/0 (socket says 172.21.15.161:47884) 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f461ffff640 1 -- 172.21.15.161:0/522321084 learned_addr learned my addr 172.21.15.161:0/522321084 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f4624e0a640 1 --2- 172.21.15.161:0/522321084 >> v2:172.21.15.183:3300/0 conn(0x7f46200d2a70 0x7f462015e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f461f7fe640 1 --2- 172.21.15.161:0/522321084 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4620162970 0x7f462015ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f461effd640 1 -- 172.21.15.161:0/522321084 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755504303 0 0) 0x7f46200d2270 con 0x7f46200d2690 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f461effd640 1 -- 172.21.15.161:0/522321084 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4608003520 con 0x7f46200d2690 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f4624e0a640 1 -- 172.21.15.161:0/522321084 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4620162970 msgr2=0x7f462015ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f4624e0a640 1 --2- 172.21.15.161:0/522321084 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4620162970 0x7f462015ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.255+0000 7f4624e0a640 1 -- 172.21.15.161:0/522321084 >> v1:172.21.15.161:6789/0 conn(0x7f46200d2690 legacy=0x7f46200d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.256+0000 7f4624e0a640 1 -- 172.21.15.161:0/522321084 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46200d0c60 con 0x7f46200d2a70 2024-03-23T14:35:34.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.256+0000 7f4624e0a640 1 --2- 172.21.15.161:0/522321084 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 0x7f462015e330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4618009d90 tx=0x7f461802bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd93b2f9214adc65 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.256+0000 7f461effd640 1 -- 172.21.15.161:0/522321084 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461802cc50 con 0x7f46200d2a70 2024-03-23T14:35:34.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.256+0000 7f461effd640 1 -- 172.21.15.161:0/522321084 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f461802cdb0 con 0x7f46200d2a70 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.256+0000 7f461effd640 1 -- 172.21.15.161:0/522321084 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618034640 con 0x7f46200d2a70 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.257+0000 7f462686d640 1 -- 172.21.15.161:0/522321084 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 msgr2=0x7f462015e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.257+0000 7f462686d640 1 --2- 172.21.15.161:0/522321084 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 0x7f462015e330 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4618009d90 tx=0x7f461802bea0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.257+0000 7f462686d640 1 -- 172.21.15.161:0/522321084 shutdown_connections 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.257+0000 7f462686d640 1 --2- 172.21.15.161:0/522321084 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 0x7f462015e330 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.257+0000 7f462686d640 1 --2- 172.21.15.161:0/522321084 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4620162970 0x7f462015ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.257+0000 7f462686d640 1 -- 172.21.15.161:0/522321084 >> 172.21.15.161:0/522321084 conn(0x7f462009d1b0 msgr2=0x7f46200d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.257+0000 7f462686d640 1 -- 172.21.15.161:0/522321084 shutdown_connections 2024-03-23T14:35:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.257+0000 7f462686d640 1 -- 172.21.15.161:0/522321084 wait complete. 2024-03-23T14:35:34.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.258+0000 7f462686d640 1 Processor -- start 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.258+0000 7f462686d640 1 -- start start 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f462686d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46200d2690 0x7f4620153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f462686d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 0x7f46200b8f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f462686d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4620162970 0x7f46200b9460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f462686d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f46200cc520 con 0x7f4620162970 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f462686d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f46200cbc80 con 0x7f46200d2a70 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f462686d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f46200cbf60 con 0x7f46200d2690 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f4624e0a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4620162970 0x7f46200b9460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f4624e0a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4620162970 0x7f46200b9460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49602/0 (socket says 172.21.15.161:49602) 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f4624e0a640 1 -- 172.21.15.161:0/646335457 learned_addr learned my addr 172.21.15.161:0/646335457 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f461ffff640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46200d2690 0x7f4620153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.259+0000 7f461f7fe640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 0x7f46200b8f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f4624e0a640 1 -- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46200d2690 msgr2=0x7f4620153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f4624e0a640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46200d2690 0x7f4620153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f4624e0a640 1 -- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 msgr2=0x7f46200b8f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f4624e0a640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 0x7f46200b8f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f4624e0a640 1 -- 172.21.15.161:0/646335457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4618009a70 con 0x7f4620162970 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f461f7fe640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 0x7f46200b8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f4624e0a640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4620162970 0x7f46200b9460 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4618009d90 tx=0x7f4618004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f461cff9640 1 -- 172.21.15.161:0/646335457 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618007000 con 0x7f4620162970 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f461cff9640 1 -- 172.21.15.161:0/646335457 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4618040670 con 0x7f4620162970 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f461cff9640 1 -- 172.21.15.161:0/646335457 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4618040a20 con 0x7f4620162970 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46200cc240 con 0x7f4620162970 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.260+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4620153780 con 0x7f4620162970 2024-03-23T14:35:34.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.261+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46200b6e40 con 0x7f4620162970 2024-03-23T14:35:34.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.261+0000 7f461cff9640 1 -- 172.21.15.161:0/646335457 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4618040b80 con 0x7f4620162970 2024-03-23T14:35:34.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.262+0000 7f461cff9640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f460c06b810 0x7f460c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.262+0000 7f461cff9640 1 -- 172.21.15.161:0/646335457 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f46180acdb0 con 0x7f4620162970 2024-03-23T14:35:34.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.262+0000 7f461ffff640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f460c06b810 0x7f460c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.262+0000 7f461ffff640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f460c06b810 0x7f460c06dcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4614005880 tx=0x7f4614009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.266+0000 7f461cff9640 1 -- 172.21.15.161:0/646335457 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f46200b6e40 con 0x7f4620162970 2024-03-23T14:35:34.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.439+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f46200b6e40 con 0x7f4620162970 2024-03-23T14:35:34.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.440+0000 7f461cff9640 1 -- 172.21.15.161:0/646335457 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f46200b6e40 con 0x7f4620162970 2024-03-23T14:35:34.436 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:34.436 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:34.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f460c06b810 msgr2=0x7f460c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f460c06b810 0x7f460c06dcd0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4614005880 tx=0x7f4614009040 comp rx=0 tx=0).stop 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4620162970 msgr2=0x7f46200b9460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4620162970 0x7f46200b9460 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4618009d90 tx=0x7f4618004a40 comp rx=0 tx=0).stop 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 shutdown_connections 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f460c06b810 0x7f460c06dcd0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4620162970 0x7f46200b9460 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46200d2a70 0x7f46200b8f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 --2- 172.21.15.161:0/646335457 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46200d2690 0x7f4620153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 >> 172.21.15.161:0/646335457 conn(0x7f462009d1b0 msgr2=0x7f46200b3960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 shutdown_connections 2024-03-23T14:35:34.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.442+0000 7f462686d640 1 -- 172.21.15.161:0/646335457 wait complete. 2024-03-23T14:35:34.449 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:34.450 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:34.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.557+0000 7f567cef4640 1 Processor -- start 2024-03-23T14:35:34.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.557+0000 7f567cef4640 1 -- start start 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f567cef4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815a2b0 0x7f567815df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f567cef4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f567806cd20 0x7f567815e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f567cef4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5678070ee0 con 0x7f567815ea00 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f567cef4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f56780b33b0 con 0x7f567806cd20 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f567cef4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f56780b3650 con 0x7f567815a2b0 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f566ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f567806cd20 0x7f567815e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f5676d9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f567815ea00 0x7f567806d6b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47900/0 (socket says 172.21.15.161:47900) 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f5676d9d640 1 -- 172.21.15.161:0/2602964816 learned_addr learned my addr 172.21.15.161:0/2602964816 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f567659c640 1 --2- 172.21.15.161:0/2602964816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815a2b0 0x7f567815df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f566ffff640 1 -- 172.21.15.161:0/2602964816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815a2b0 msgr2=0x7f567815df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:34.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f566ffff640 1 --2- 172.21.15.161:0/2602964816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815a2b0 0x7f567815df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f566ffff640 1 -- 172.21.15.161:0/2602964816 >> v1:172.21.15.161:6789/0 conn(0x7f567815ea00 legacy=0x7f567806d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f5675d9b640 1 -- 172.21.15.161:0/2602964816 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 924612583 0 0) 0x7f5678070ee0 con 0x7f567815ea00 2024-03-23T14:35:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f566ffff640 1 -- 172.21.15.161:0/2602964816 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56780b3930 con 0x7f567806cd20 2024-03-23T14:35:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.558+0000 7f567659c640 1 --2- 172.21.15.161:0/2602964816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815a2b0 0x7f567815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.559+0000 7f566ffff640 1 --2- 172.21.15.161:0/2602964816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567806cd20 0x7f567815e4c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56600097a0 tx=0x7f566002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40dc078a57771504 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.559+0000 7f5675d9b640 1 -- 172.21.15.161:0/2602964816 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660031830 con 0x7f567806cd20 2024-03-23T14:35:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.559+0000 7f5675d9b640 1 -- 172.21.15.161:0/2602964816 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5660031990 con 0x7f567806cd20 2024-03-23T14:35:34.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.559+0000 7f5675d9b640 1 -- 172.21.15.161:0/2602964816 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660031d20 con 0x7f567806cd20 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.560+0000 7f567cef4640 1 -- 172.21.15.161:0/2602964816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567806cd20 msgr2=0x7f567815e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.560+0000 7f567cef4640 1 --2- 172.21.15.161:0/2602964816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567806cd20 0x7f567815e4c0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f56600097a0 tx=0x7f566002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.560+0000 7f567cef4640 1 -- 172.21.15.161:0/2602964816 shutdown_connections 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.560+0000 7f567cef4640 1 --2- 172.21.15.161:0/2602964816 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567806cd20 0x7f567815e4c0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.560+0000 7f567cef4640 1 --2- 172.21.15.161:0/2602964816 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815a2b0 0x7f567815df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.560+0000 7f567cef4640 1 -- 172.21.15.161:0/2602964816 >> 172.21.15.161:0/2602964816 conn(0x7f567806b260 msgr2=0x7f567806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.560+0000 7f567cef4640 1 -- 172.21.15.161:0/2602964816 shutdown_connections 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.560+0000 7f567cef4640 1 -- 172.21.15.161:0/2602964816 wait complete. 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.561+0000 7f567cef4640 1 Processor -- start 2024-03-23T14:35:34.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.561+0000 7f567cef4640 1 -- start start 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f567cef4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f567806cd20 0x7f5678159090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f567cef4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567815a2b0 0x7f56780b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f567cef4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815ea00 0x7f56780b1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f567cef4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f56780a95d0 con 0x7f567815a2b0 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f567cef4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f56781595d0 con 0x7f567806cd20 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f567cef4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f56780ae950 con 0x7f567815ea00 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567815a2b0 0x7f56780b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567815a2b0 0x7f56780b0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49618/0 (socket says 172.21.15.161:49618) 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 -- 172.21.15.161:0/1932654028 learned_addr learned my addr 172.21.15.161:0/1932654028 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f567659c640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f567806cd20 0x7f5678159090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f5676d9d640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815ea00 0x7f56780b1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 -- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815ea00 msgr2=0x7f56780b1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815ea00 0x7f56780b1380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 -- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f567806cd20 msgr2=0x7f5678159090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f567806cd20 0x7f5678159090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 -- 172.21.15.161:0/1932654028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5668009400 con 0x7f567815a2b0 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566ffff640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567815a2b0 0x7f56780b0e40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5660000c00 tx=0x7f56600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566effd640 1 -- 172.21.15.161:0/1932654028 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5660004230 con 0x7f567815a2b0 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566effd640 1 -- 172.21.15.161:0/1932654028 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5660004390 con 0x7f567815a2b0 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f566effd640 1 -- 172.21.15.161:0/1932654028 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566003a880 con 0x7f567815a2b0 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.562+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5660009400 con 0x7f567815a2b0 2024-03-23T14:35:34.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.563+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56780aef10 con 0x7f567815a2b0 2024-03-23T14:35:34.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.564+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f567804f4e0 con 0x7f567815a2b0 2024-03-23T14:35:34.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.564+0000 7f566effd640 1 -- 172.21.15.161:0/1932654028 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f566003a9e0 con 0x7f567815a2b0 2024-03-23T14:35:34.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.564+0000 7f566effd640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f565806b810 0x7f565806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.564+0000 7f566effd640 1 -- 172.21.15.161:0/1932654028 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f56600ac5a0 con 0x7f567815a2b0 2024-03-23T14:35:34.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.565+0000 7f567659c640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f565806b810 0x7f565806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.565+0000 7f567659c640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f565806b810 0x7f565806dcd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5668009cd0 tx=0x7f5668009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.569+0000 7f566effd640 1 -- 172.21.15.161:0/1932654028 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5660077190 con 0x7f567815a2b0 2024-03-23T14:35:34.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.739+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f56780ce520 con 0x7f567815a2b0 2024-03-23T14:35:34.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.740+0000 7f566effd640 1 -- 172.21.15.161:0/1932654028 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f566007a850 con 0x7f567815a2b0 2024-03-23T14:35:34.736 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f565806b810 msgr2=0x7f565806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f565806b810 0x7f565806dcd0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5668009cd0 tx=0x7f5668009190 comp rx=0 tx=0).stop 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567815a2b0 msgr2=0x7f56780b0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567815a2b0 0x7f56780b0e40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f5660000c00 tx=0x7f56600046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 shutdown_connections 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f565806b810 0x7f565806dcd0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f567815ea00 0x7f56780b1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f567815a2b0 0x7f56780b0e40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 --2- 172.21.15.161:0/1932654028 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f567806cd20 0x7f5678159090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 >> 172.21.15.161:0/1932654028 conn(0x7f567806b260 msgr2=0x7f5678154870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:34.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 shutdown_connections 2024-03-23T14:35:34.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.742+0000 7f567cef4640 1 -- 172.21.15.161:0/1932654028 wait complete. 2024-03-23T14:35:34.748 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:34.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.857+0000 7fc9a842a640 1 Processor -- start 2024-03-23T14:35:34.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.857+0000 7fc9a842a640 1 -- start start 2024-03-23T14:35:34.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.858+0000 7fc9a842a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9a0161750 0x7fc9a00d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.858+0000 7fc9a842a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a00d1330 con 0x7fc9a00d1cb0 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.858+0000 7fc9a842a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc9a0162e80 con 0x7fc9a0161b30 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.858+0000 7fc9a842a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc9a00d0720 con 0x7fc9a0161750 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.859+0000 7fc9a61c6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9a0161750 0x7fc9a00d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.859+0000 7fc9a59c5640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc9a00d1cb0 0x7fc9a0162060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47908/0 (socket says 172.21.15.161:47908) 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.859+0000 7fc9a59c5640 1 -- 172.21.15.161:0/896932037 learned_addr learned my addr 172.21.15.161:0/896932037 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.859+0000 7fc9a61c6640 1 -- 172.21.15.161:0/896932037 >> v1:172.21.15.183:6789/0 conn(0x7fc9a0161b30 legacy=0x7fc9a0162770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.859+0000 7fc9a51c4640 1 -- 172.21.15.161:0/896932037 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150439204 0 0) 0x7fc9a0162e80 con 0x7fc9a0161b30 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.859+0000 7fc9a61c6640 1 -- 172.21.15.161:0/896932037 >> v1:172.21.15.161:6789/0 conn(0x7fc9a00d1cb0 legacy=0x7fc9a0162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.859+0000 7fc9a61c6640 1 -- 172.21.15.161:0/896932037 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9a00d0a00 con 0x7fc9a0161750 2024-03-23T14:35:34.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a61c6640 1 --2- 172.21.15.161:0/896932037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00d25c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0097a0 tx=0x7fc99c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c99b5a3c461828a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a51c4640 1 -- 172.21.15.161:0/896932037 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c030710 con 0x7fc9a0161750 2024-03-23T14:35:34.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a51c4640 1 -- 172.21.15.161:0/896932037 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc99c02e050 con 0x7fc9a0161750 2024-03-23T14:35:34.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a51c4640 1 -- 172.21.15.161:0/896932037 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc99c030a90 con 0x7fc9a0161750 2024-03-23T14:35:34.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a842a640 1 -- 172.21.15.161:0/896932037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 msgr2=0x7fc9a00d25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a842a640 1 --2- 172.21.15.161:0/896932037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00d25c0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc99c0097a0 tx=0x7fc99c02b790 comp rx=0 tx=0).stop 2024-03-23T14:35:34.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a842a640 1 -- 172.21.15.161:0/896932037 shutdown_connections 2024-03-23T14:35:34.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a842a640 1 --2- 172.21.15.161:0/896932037 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00d25c0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a842a640 1 -- 172.21.15.161:0/896932037 >> 172.21.15.161:0/896932037 conn(0x7fc9a009d450 msgr2=0x7fc9a00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:34.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a842a640 1 -- 172.21.15.161:0/896932037 shutdown_connections 2024-03-23T14:35:34.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.860+0000 7fc9a842a640 1 -- 172.21.15.161:0/896932037 wait complete. 2024-03-23T14:35:34.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a842a640 1 Processor -- start 2024-03-23T14:35:34.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a842a640 1 -- start start 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a842a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9a00d1cb0 0x7fc9a01f7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a842a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a842a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9a0161b30 0x7fc9a015c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a842a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a01f8560 con 0x7fc9a0161750 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a842a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc9a01f4060 con 0x7fc9a0161b30 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a842a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc9a00cdf00 con 0x7fc9a00d1cb0 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a59c5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a59c5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00ca4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49634/0 (socket says 172.21.15.161:49634) 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a59c5640 1 -- 172.21.15.161:0/2729023373 learned_addr learned my addr 172.21.15.161:0/2729023373 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:34.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a69c7640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9a0161b30 0x7fc9a015c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.862+0000 7fc9a61c6640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9a00d1cb0 0x7fc9a01f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc9a59c5640 1 -- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9a00d1cb0 msgr2=0x7fc9a01f7630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc9a59c5640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9a00d1cb0 0x7fc9a01f7630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc9a59c5640 1 -- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9a0161b30 msgr2=0x7fc9a015c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc9a59c5640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9a0161b30 0x7fc9a015c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc9a59c5640 1 -- 172.21.15.161:0/2729023373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc99c009400 con 0x7fc9a0161750 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc9a59c5640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00ca4f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc99000c780 tx=0x7fc99000cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc996ffd640 1 -- 172.21.15.161:0/2729023373 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9900072d0 con 0x7fc9a0161750 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc996ffd640 1 -- 172.21.15.161:0/2729023373 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc990007430 con 0x7fc9a0161750 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9a01f7cc0 con 0x7fc9a0161750 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc996ffd640 1 -- 172.21.15.161:0/2729023373 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9900047e0 con 0x7fc9a0161750 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.863+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9a01f8280 con 0x7fc9a0161750 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.864+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9a0072c70 con 0x7fc9a0161750 2024-03-23T14:35:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.864+0000 7fc996ffd640 1 -- 172.21.15.161:0/2729023373 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc990004960 con 0x7fc9a0161750 2024-03-23T14:35:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.865+0000 7fc996ffd640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc98406b840 0x7fc98406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.865+0000 7fc996ffd640 1 -- 172.21.15.161:0/2729023373 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc990089780 con 0x7fc9a0161750 2024-03-23T14:35:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.865+0000 7fc9a61c6640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc98406b840 0x7fc98406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.865+0000 7fc9a61c6640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc98406b840 0x7fc98406dd00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc99c034000 tx=0x7fc99c02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:34.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:34.869+0000 7fc996ffd640 1 -- 172.21.15.161:0/2729023373 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc990054250 con 0x7fc9a0161750 2024-03-23T14:35:35.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.039+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fc9a00a2030 con 0x7fc9a0161750 2024-03-23T14:35:35.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.039+0000 7fc996ffd640 1 -- 172.21.15.161:0/2729023373 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fc990057910 con 0x7fc9a0161750 2024-03-23T14:35:35.036 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:35.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc98406b840 msgr2=0x7fc98406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc98406b840 0x7fc98406dd00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc99c034000 tx=0x7fc99c02f000 comp rx=0 tx=0).stop 2024-03-23T14:35:35.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 msgr2=0x7fc9a00ca4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00ca4f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fc99000c780 tx=0x7fc99000cc40 comp rx=0 tx=0).stop 2024-03-23T14:35:35.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 shutdown_connections 2024-03-23T14:35:35.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc98406b840 0x7fc98406dd00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9a0161b30 0x7fc9a015c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9a0161750 0x7fc9a00ca4f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 --2- 172.21.15.161:0/2729023373 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9a00d1cb0 0x7fc9a01f7630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 >> 172.21.15.161:0/2729023373 conn(0x7fc9a009d450 msgr2=0x7fc9a006c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:35.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 shutdown_connections 2024-03-23T14:35:35.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.041+0000 7fc9a842a640 1 -- 172.21.15.161:0/2729023373 wait complete. 2024-03-23T14:35:35.048 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:35.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.154+0000 7f2890b3e640 1 Processor -- start 2024-03-23T14:35:35.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.155+0000 7f2890b3e640 1 -- start start 2024-03-23T14:35:35.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.155+0000 7f2890b3e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f288c161820 0x7f288c0d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.155+0000 7f2890b3e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f288c0d1270 0x7f288c0d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.155+0000 7f2890b3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f288c161c00 con 0x7f288c0d29d0 2024-03-23T14:35:35.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.155+0000 7f2890b3e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f288c0d2e20 con 0x7f288c0d1270 2024-03-23T14:35:35.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.155+0000 7f2890b3e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f288c0cea30 con 0x7f288c161820 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.155+0000 7f2883fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f288c0d1270 0x7f288c0d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.156+0000 7f2883fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f288c0d1270 0x7f288c0d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49638/0 (socket says 172.21.15.161:49638) 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.156+0000 7f2883fff640 1 -- 172.21.15.161:0/750164972 learned_addr learned my addr 172.21.15.161:0/750164972 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.156+0000 7f288a59c640 1 --2- 172.21.15.161:0/750164972 >> v2:172.21.15.183:3300/0 conn(0x7f288c161820 0x7f288c0d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.156+0000 7f2883fff640 1 -- 172.21.15.161:0/750164972 >> v2:172.21.15.183:3300/0 conn(0x7f288c161820 msgr2=0x7f288c0d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.156+0000 7f2883fff640 1 --2- 172.21.15.161:0/750164972 >> v2:172.21.15.183:3300/0 conn(0x7f288c161820 0x7f288c0d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.156+0000 7f2883fff640 1 -- 172.21.15.161:0/750164972 >> v1:172.21.15.183:6789/0 conn(0x7f288c0d29d0 legacy=0x7f288c161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.156+0000 7f2883fff640 1 -- 172.21.15.161:0/750164972 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f288c0cfd70 con 0x7f288c0d1270 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.156+0000 7f2883fff640 1 --2- 172.21.15.161:0/750164972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c0d2490 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f288c06b270 tx=0x7f287402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8000e9a177d33a61 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2889d9b640 1 -- 172.21.15.161:0/750164972 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874030860 con 0x7f288c0d1270 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2889d9b640 1 -- 172.21.15.161:0/750164972 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28740309c0 con 0x7f288c0d1270 2024-03-23T14:35:35.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2889d9b640 1 -- 172.21.15.161:0/750164972 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2874030d50 con 0x7f288c0d1270 2024-03-23T14:35:35.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2890b3e640 1 -- 172.21.15.161:0/750164972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 msgr2=0x7f288c0d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2890b3e640 1 --2- 172.21.15.161:0/750164972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c0d2490 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f288c06b270 tx=0x7f287402b9a0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.154 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2890b3e640 1 -- 172.21.15.161:0/750164972 shutdown_connections 2024-03-23T14:35:35.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2890b3e640 1 --2- 172.21.15.161:0/750164972 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c0d2490 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2890b3e640 1 --2- 172.21.15.161:0/750164972 >> v2:172.21.15.183:3300/0 conn(0x7f288c161820 0x7f288c0d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2890b3e640 1 -- 172.21.15.161:0/750164972 >> 172.21.15.161:0/750164972 conn(0x7f288c09d190 msgr2=0x7f288c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:35.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2890b3e640 1 -- 172.21.15.161:0/750164972 shutdown_connections 2024-03-23T14:35:35.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.157+0000 7f2890b3e640 1 -- 172.21.15.161:0/750164972 wait complete. 2024-03-23T14:35:35.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f2890b3e640 1 Processor -- start 2024-03-23T14:35:35.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f2890b3e640 1 -- start start 2024-03-23T14:35:35.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f2890b3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c15e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f2890b3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f288c0d29d0 0x7f288c0b9330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f2890b3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f288c161820 0x7f288c0b9870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f2890b3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f288c15c380 con 0x7f288c0d1270 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f2890b3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f288c0c9930 con 0x7f288c0d29d0 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f2890b3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f288c0c9b80 con 0x7f288c161820 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f288a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c15e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f288a59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c15e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49642/0 (socket says 172.21.15.161:49642) 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.159+0000 7f288a59c640 1 -- 172.21.15.161:0/1694393739 learned_addr learned my addr 172.21.15.161:0/1694393739 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f2883fff640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f288c0d29d0 0x7f288c0b9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f288ad9d640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f288c161820 0x7f288c0b9870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f288a59c640 1 -- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f288c161820 msgr2=0x7f288c0b9870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f288a59c640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f288c161820 0x7f288c0b9870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f288a59c640 1 -- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f288c0d29d0 msgr2=0x7f288c0b9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f288a59c640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f288c0d29d0 0x7f288c0b9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f288a59c640 1 -- 172.21.15.161:0/1694393739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28740094d0 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f288ad9d640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f288c161820 0x7f288c0b9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f288a59c640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c15e430 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f287c00e9b0 tx=0x7f287c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f2882ffd640 1 -- 172.21.15.161:0/1694393739 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c009670 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f2882ffd640 1 -- 172.21.15.161:0/1694393739 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f287c00c9a0 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f288c0c9e60 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f2882ffd640 1 -- 172.21.15.161:0/1694393739 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f287c00cd50 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.160+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f288c0ca420 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.161+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f288c04f4e0 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.161+0000 7f2882ffd640 1 -- 172.21.15.161:0/1694393739 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f287c013470 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.162+0000 7f2882ffd640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f286c06b810 0x7f286c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.162+0000 7f2882ffd640 1 -- 172.21.15.161:0/1694393739 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f287c08a370 con 0x7f288c0d1270 2024-03-23T14:35:35.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.162+0000 7f2883fff640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f286c06b810 0x7f286c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.162+0000 7f2883fff640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f286c06b810 0x7f286c06dcd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2874036000 tx=0x7f2874008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.166+0000 7f2882ffd640 1 -- 172.21.15.161:0/1694393739 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f287c054e40 con 0x7f288c0d1270 2024-03-23T14:35:35.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.341+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f288c0d2270 con 0x7f288c0d1270 2024-03-23T14:35:35.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.342+0000 7f2882ffd640 1 -- 172.21.15.161:0/1694393739 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f287c058500 con 0x7f288c0d1270 2024-03-23T14:35:35.338 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f286c06b810 msgr2=0x7f286c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f286c06b810 0x7f286c06dcd0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2874036000 tx=0x7f2874008030 comp rx=0 tx=0).stop 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 msgr2=0x7f288c15e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c15e430 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f287c00e9b0 tx=0x7f287c00ee70 comp rx=0 tx=0).stop 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 shutdown_connections 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f286c06b810 0x7f286c06dcd0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f288c161820 0x7f288c0b9870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f288c0d29d0 0x7f288c0b9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 --2- 172.21.15.161:0/1694393739 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f288c0d1270 0x7f288c15e430 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 >> 172.21.15.161:0/1694393739 conn(0x7f288c09d190 msgr2=0x7f288c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:35.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 shutdown_connections 2024-03-23T14:35:35.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.344+0000 7f2890b3e640 1 -- 172.21.15.161:0/1694393739 wait complete. 2024-03-23T14:35:35.350 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:35.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.456+0000 7fae0533e640 1 Processor -- start 2024-03-23T14:35:35.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.456+0000 7fae0533e640 1 -- start start 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fae0533e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fae0015e400 0x7fae000cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fae0533e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fae000d2b40 0x7fae000b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fae0533e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae000d0460 con 0x7fae000cf9d0 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fae0533e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fae000d1050 con 0x7fae0015e400 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fae0533e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fae000d06b0 con 0x7fae000d2b40 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadfeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fae0015e400 0x7fae000cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadfeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fae0015e400 0x7fae000cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49656/0 (socket says 172.21.15.161:49656) 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadfeffd640 1 -- 172.21.15.161:0/2477637301 learned_addr learned my addr 172.21.15.161:0/2477637301 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadff7fe640 1 --2- 172.21.15.161:0/2477637301 >> v2:172.21.15.183:3300/0 conn(0x7fae000d2b40 0x7fae000b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadff7fe640 1 -- 172.21.15.161:0/2477637301 >> v2:172.21.15.161:3300/0 conn(0x7fae0015e400 msgr2=0x7fae000cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadff7fe640 1 --2- 172.21.15.161:0/2477637301 >> v2:172.21.15.161:3300/0 conn(0x7fae0015e400 0x7fae000cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadff7fe640 1 -- 172.21.15.161:0/2477637301 >> v1:172.21.15.183:6789/0 conn(0x7fae000cf9d0 legacy=0x7fae000d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadff7fe640 1 -- 172.21.15.161:0/2477637301 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae000d0990 con 0x7fae000d2b40 2024-03-23T14:35:35.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.457+0000 7fadfeffd640 1 --2- 172.21.15.161:0/2477637301 >> v2:172.21.15.161:3300/0 conn(0x7fae0015e400 0x7fae000cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:35.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.458+0000 7fadff7fe640 1 --2- 172.21.15.161:0/2477637301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 0x7fae000b0a80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fae000ce430 tx=0x7fadf802bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65fe568021ca31dd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.458+0000 7fadfe7fc640 1 -- 172.21.15.161:0/2477637301 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf802cc80 con 0x7fae000d2b40 2024-03-23T14:35:35.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.458+0000 7fadfe7fc640 1 -- 172.21.15.161:0/2477637301 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf802cde0 con 0x7fae000d2b40 2024-03-23T14:35:35.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.458+0000 7fadfe7fc640 1 -- 172.21.15.161:0/2477637301 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf8034660 con 0x7fae000d2b40 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.458+0000 7fae0533e640 1 -- 172.21.15.161:0/2477637301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 msgr2=0x7fae000b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.459+0000 7fae0533e640 1 --2- 172.21.15.161:0/2477637301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 0x7fae000b0a80 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fae000ce430 tx=0x7fadf802bed0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.459+0000 7fae0533e640 1 -- 172.21.15.161:0/2477637301 shutdown_connections 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.459+0000 7fae0533e640 1 --2- 172.21.15.161:0/2477637301 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 0x7fae000b0a80 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.459+0000 7fae0533e640 1 --2- 172.21.15.161:0/2477637301 >> v2:172.21.15.161:3300/0 conn(0x7fae0015e400 0x7fae000cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.459+0000 7fae0533e640 1 -- 172.21.15.161:0/2477637301 >> 172.21.15.161:0/2477637301 conn(0x7fae0009d190 msgr2=0x7fae000d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.459+0000 7fae0533e640 1 -- 172.21.15.161:0/2477637301 shutdown_connections 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.459+0000 7fae0533e640 1 -- 172.21.15.161:0/2477637301 wait complete. 2024-03-23T14:35:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.460+0000 7fae0533e640 1 Processor -- start 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.460+0000 7fae0533e640 1 -- start start 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.460+0000 7fae0533e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae000cf9d0 0x7fae001f4810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.460+0000 7fae0533e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 0x7fae0015ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.460+0000 7fae0533e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae0015e400 0x7fae0015cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.460+0000 7fae0533e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fae001f4df0 con 0x7fae000cf9d0 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.460+0000 7fae0533e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fae000c7b20 con 0x7fae000d2b40 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.460+0000 7fae0533e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fae000c7e00 con 0x7fae0015e400 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae000cf9d0 0x7fae001f4810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae000cf9d0 0x7fae001f4810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49660/0 (socket says 172.21.15.161:49660) 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 -- 172.21.15.161:0/3325344724 learned_addr learned my addr 172.21.15.161:0/3325344724 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadf7fff640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 0x7fae0015ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadff7fe640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae0015e400 0x7fae0015cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 -- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae0015e400 msgr2=0x7fae0015cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae0015e400 0x7fae0015cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 -- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 msgr2=0x7fae0015ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 0x7fae0015ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 -- 172.21.15.161:0/3325344724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf00094d0 con 0x7fae000cf9d0 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadf7fff640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 0x7fae0015ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadfeffd640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae000cf9d0 0x7fae001f4810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fadf000e9b0 tx=0x7fadf000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadf77fe640 1 -- 172.21.15.161:0/3325344724 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf000c9a0 con 0x7fae000cf9d0 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadf77fe640 1 -- 172.21.15.161:0/3325344724 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fadf000cb00 con 0x7fae000cf9d0 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf8009a70 con 0x7fae000cf9d0 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fadf77fe640 1 -- 172.21.15.161:0/3325344724 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf000ce70 con 0x7fae000cf9d0 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.461+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae000c83c0 con 0x7fae000cf9d0 2024-03-23T14:35:35.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.462+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae000a1e00 con 0x7fae000cf9d0 2024-03-23T14:35:35.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.463+0000 7fadf77fe640 1 -- 172.21.15.161:0/3325344724 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fadf0013470 con 0x7fae000cf9d0 2024-03-23T14:35:35.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.463+0000 7fadf77fe640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fade006b810 0x7fade006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.463+0000 7fadf77fe640 1 -- 172.21.15.161:0/3325344724 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fadf008a790 con 0x7fae000cf9d0 2024-03-23T14:35:35.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.463+0000 7fadf7fff640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fade006b810 0x7fade006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.464+0000 7fadf7fff640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fade006b810 0x7fade006dcd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fae000d3ea0 tx=0x7fade8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.467+0000 7fadf77fe640 1 -- 172.21.15.161:0/3325344724 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fadf00551d0 con 0x7fae000cf9d0 2024-03-23T14:35:35.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.641+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fae0004f550 con 0x7fae000cf9d0 2024-03-23T14:35:35.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.643+0000 7fadf77fe640 1 -- 172.21.15.161:0/3325344724 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7fadf0017090 con 0x7fae000cf9d0 2024-03-23T14:35:35.638 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:35.639 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fade006b810 msgr2=0x7fade006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fade006b810 0x7fade006dcd0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fae000d3ea0 tx=0x7fade8009040 comp rx=0 tx=0).stop 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae000cf9d0 msgr2=0x7fae001f4810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae000cf9d0 0x7fae001f4810 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fadf000e9b0 tx=0x7fadf000ee70 comp rx=0 tx=0).stop 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 shutdown_connections 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fade006b810 0x7fade006dcd0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae0015e400 0x7fae0015cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae000d2b40 0x7fae0015ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 --2- 172.21.15.161:0/3325344724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae000cf9d0 0x7fae001f4810 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 >> 172.21.15.161:0/3325344724 conn(0x7fae0009d190 msgr2=0x7fae000b10b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:35.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 shutdown_connections 2024-03-23T14:35:35.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.645+0000 7fae0533e640 1 -- 172.21.15.161:0/3325344724 wait complete. 2024-03-23T14:35:35.652 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:35.653 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:35.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.761+0000 7ffafc5f4640 1 Processor -- start 2024-03-23T14:35:35.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.761+0000 7ffafc5f4640 1 -- start start 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffafc5f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffaf40ce930 0x7ffaf40cb180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffafc5f4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaf40cbad0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffafc5f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaf40cedc0 con 0x7ffaf415e080 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffafc5f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffaf4161b40 con 0x7ffaf40ce930 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffaf9b8f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffaf40ce930 0x7ffaf40cb180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffafab91640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ffaf40cb6c0 0x7ffaf4071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47920/0 (socket says 172.21.15.161:47920) 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffafab91640 1 -- 172.21.15.161:0/3769073395 learned_addr learned my addr 172.21.15.161:0/3769073395 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffaf9b8f640 1 -- 172.21.15.161:0/3769073395 >> v1:172.21.15.183:6789/0 conn(0x7ffaf415e080 legacy=0x7ffaf40afaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffaf9b8f640 1 -- 172.21.15.161:0/3769073395 >> v1:172.21.15.161:6789/0 conn(0x7ffaf40cb6c0 legacy=0x7ffaf4071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffaf938e640 1 -- 172.21.15.161:0/3769073395 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2449637711 0 0) 0x7ffaf40cbad0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.762+0000 7ffaf9b8f640 1 -- 172.21.15.161:0/3769073395 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaf4161d50 con 0x7ffaf40ce930 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.763+0000 7ffaf9b8f640 1 --2- 172.21.15.161:0/3769073395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40ce930 0x7ffaf40cb180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffae40097a0 tx=0x7ffae402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=938a2ab2d9cc53a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.763+0000 7ffaf938e640 1 -- 172.21.15.161:0/3769073395 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4030610 con 0x7ffaf40ce930 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.763+0000 7ffaf938e640 1 -- 172.21.15.161:0/3769073395 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffae4030770 con 0x7ffaf40ce930 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.763+0000 7ffaf938e640 1 -- 172.21.15.161:0/3769073395 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae4030b00 con 0x7ffaf40ce930 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.763+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3769073395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40ce930 msgr2=0x7ffaf40cb180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.763+0000 7ffafc5f4640 1 --2- 172.21.15.161:0/3769073395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40ce930 0x7ffaf40cb180 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7ffae40097a0 tx=0x7ffae402b960 comp rx=0 tx=0).stop 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.764+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3769073395 shutdown_connections 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.764+0000 7ffafc5f4640 1 --2- 172.21.15.161:0/3769073395 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40ce930 0x7ffaf40cb180 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.764+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3769073395 >> 172.21.15.161:0/3769073395 conn(0x7ffaf40d0b30 msgr2=0x7ffaf40d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.764+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3769073395 shutdown_connections 2024-03-23T14:35:35.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.764+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3769073395 wait complete. 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.765+0000 7ffafc5f4640 1 Processor -- start 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.765+0000 7ffafc5f4640 1 -- start start 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.765+0000 7ffafc5f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40cb6c0 0x7ffaf40ae540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.765+0000 7ffafc5f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf40ce930 0x7ffaf40cddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.765+0000 7ffafc5f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf415e080 0x7ffaf40ce2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafc5f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf414f5c0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafc5f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffaf414f2e0 con 0x7ffaf40ce930 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafc5f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ffaf40ada60 con 0x7ffaf415e080 2024-03-23T14:35:35.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40cb6c0 0x7ffaf40ae540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40cb6c0 0x7ffaf40ae540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49674/0 (socket says 172.21.15.161:49674) 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 -- 172.21.15.161:0/3161247982 learned_addr learned my addr 172.21.15.161:0/3161247982 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffaf9b8f640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf40ce930 0x7ffaf40cddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafab91640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf415e080 0x7ffaf40ce2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 -- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf415e080 msgr2=0x7ffaf40ce2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf415e080 0x7ffaf40ce2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 -- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf40ce930 msgr2=0x7ffaf40cddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf40ce930 0x7ffaf40cddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 -- 172.21.15.161:0/3161247982 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae4009400 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafa390640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40cb6c0 0x7ffaf40ae540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000c450 tx=0x7ffaf000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffaeaffd640 1 -- 172.21.15.161:0/3161247982 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf00052d0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffaeaffd640 1 -- 172.21.15.161:0/3161247982 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf0005430 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaf40add40 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffaeaffd640 1 -- 172.21.15.161:0/3161247982 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0012870 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.766+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaf414c060 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.767+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaf40ad8c0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.768+0000 7ffaeaffd640 1 -- 172.21.15.161:0/3161247982 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffaf00129d0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.768+0000 7ffaeaffd640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad806b810 0x7ffad806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:35.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.768+0000 7ffaeaffd640 1 -- 172.21.15.161:0/3161247982 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ffaf0089bb0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.768+0000 7ffaf9b8f640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad806b810 0x7ffad806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:35.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.769+0000 7ffaf9b8f640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad806b810 0x7ffad806dcd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffae4036000 tx=0x7ffae4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:35.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.772+0000 7ffaeaffd640 1 -- 172.21.15.161:0/3161247982 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffaf40ad8c0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.943+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffaf40ad8c0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.944+0000 7ffaeaffd640 1 -- 172.21.15.161:0/3161247982 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7ffaf40ad8c0 con 0x7ffaf40cb6c0 2024-03-23T14:35:35.940 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:35.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad806b810 msgr2=0x7ffad806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad806b810 0x7ffad806dcd0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ffae4036000 tx=0x7ffae4008030 comp rx=0 tx=0).stop 2024-03-23T14:35:35.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40cb6c0 msgr2=0x7ffaf40ae540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:35.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40cb6c0 0x7ffaf40ae540 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000c450 tx=0x7ffaf000c910 comp rx=0 tx=0).stop 2024-03-23T14:35:35.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 shutdown_connections 2024-03-23T14:35:35.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad806b810 0x7ffad806dcd0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffaf415e080 0x7ffaf40ce2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffaf40ce930 0x7ffaf40cddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 --2- 172.21.15.161:0/3161247982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffaf40cb6c0 0x7ffaf40ae540 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:35.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 >> 172.21.15.161:0/3161247982 conn(0x7ffaf40d0b30 msgr2=0x7ffaf406e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:35.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 shutdown_connections 2024-03-23T14:35:35.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:35.946+0000 7ffafc5f4640 1 -- 172.21.15.161:0/3161247982 wait complete. 2024-03-23T14:35:35.952 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:36.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.060+0000 7f4c2c66b640 1 Processor -- start 2024-03-23T14:35:36.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.060+0000 7f4c2c66b640 1 -- start start 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c2c66b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2406e950 0x7f4c2415b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c2c66b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4c2415b5b0 0x7f4c2406d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c2c66b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c2415ba40 con 0x7f4c241549c0 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c2c66b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f4c240b2a10 con 0x7f4c2415b5b0 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c2c66b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4c24071330 con 0x7f4c2406e950 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c29c06640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2406e950 0x7f4c2415b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c2ac08640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4c2415b5b0 0x7f4c2406d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c29c06640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2406e950 0x7f4c2415b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:47358/0 (socket says 172.21.15.161:47358) 2024-03-23T14:35:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.061+0000 7f4c29c06640 1 -- 172.21.15.161:0/3766992089 learned_addr learned my addr 172.21.15.161:0/3766992089 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.062+0000 7f4c29c06640 1 -- 172.21.15.161:0/3766992089 >> v2:172.21.15.183:3300/0 conn(0x7f4c2415b5b0 msgr2=0x7f4c2406d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.062+0000 7f4c29c06640 1 --2- 172.21.15.161:0/3766992089 >> v2:172.21.15.183:3300/0 conn(0x7f4c2415b5b0 0x7f4c2406d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.062+0000 7f4c29c06640 1 -- 172.21.15.161:0/3766992089 >> v1:172.21.15.183:6789/0 conn(0x7f4c241549c0 legacy=0x7f4c24070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.062+0000 7f4c29c06640 1 -- 172.21.15.161:0/3766992089 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c2406edd0 con 0x7f4c2406e950 2024-03-23T14:35:36.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.062+0000 7f4c29405640 1 -- 172.21.15.161:0/3766992089 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2957833771 0 0) 0x7f4c20001490 con 0x7f4c241549c0 2024-03-23T14:35:36.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.062+0000 7f4c2ac08640 1 --2- 172.21.15.161:0/3766992089 >> v2:172.21.15.183:3300/0 conn(0x7f4c2415b5b0 0x7f4c2406d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:36.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.062+0000 7f4c29c06640 1 --2- 172.21.15.161:0/3766992089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2406e950 0x7f4c2415b070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009870 tx=0x7f4c1402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79686a6906a44221 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c29405640 1 -- 172.21.15.161:0/3766992089 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14030840 con 0x7f4c2406e950 2024-03-23T14:35:36.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c29405640 1 -- 172.21.15.161:0/3766992089 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c140309a0 con 0x7f4c2406e950 2024-03-23T14:35:36.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c29405640 1 -- 172.21.15.161:0/3766992089 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14030d30 con 0x7f4c2406e950 2024-03-23T14:35:36.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3766992089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2406e950 msgr2=0x7f4c2415b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3766992089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2406e950 0x7f4c2415b070 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009870 tx=0x7f4c1402b980 comp rx=0 tx=0).stop 2024-03-23T14:35:36.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3766992089 shutdown_connections 2024-03-23T14:35:36.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3766992089 >> v2:172.21.15.183:3300/0 conn(0x7f4c2415b5b0 0x7f4c2406d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3766992089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2406e950 0x7f4c2415b070 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3766992089 >> 172.21.15.161:0/3766992089 conn(0x7f4c2406b4a0 msgr2=0x7f4c2406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:36.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3766992089 shutdown_connections 2024-03-23T14:35:36.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.063+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3766992089 wait complete. 2024-03-23T14:35:36.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.064+0000 7f4c2c66b640 1 Processor -- start 2024-03-23T14:35:36.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.064+0000 7f4c2c66b640 1 -- start start 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c2c66b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c2406e950 0x7f4c241f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c2c66b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c241549c0 0x7f4c24159230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c2c66b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2415b5b0 0x7f4c2414dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c2c66b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4c24159c80 con 0x7f4c241549c0 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c2c66b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4c24159770 con 0x7f4c2406e950 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c2c66b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4c2414e400 con 0x7f4c2415b5b0 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c29c06640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c241549c0 0x7f4c24159230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c29c06640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c241549c0 0x7f4c24159230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49686/0 (socket says 172.21.15.161:49686) 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c29c06640 1 -- 172.21.15.161:0/3872616842 learned_addr learned my addr 172.21.15.161:0/3872616842 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:36.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c2a407640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c2406e950 0x7f4c241f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.065+0000 7f4c2ac08640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2415b5b0 0x7f4c2414dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c29c06640 1 -- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2415b5b0 msgr2=0x7f4c2414dec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c29c06640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2415b5b0 0x7f4c2414dec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c29c06640 1 -- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c2406e950 msgr2=0x7f4c241f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c29c06640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c2406e950 0x7f4c241f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c29c06640 1 -- 172.21.15.161:0/3872616842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c1c00a5a0 con 0x7f4c241549c0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c2ac08640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2415b5b0 0x7f4c2414dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c29c06640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c241549c0 0x7f4c24159230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009870 tx=0x7f4c14004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c1affd640 1 -- 172.21.15.161:0/3872616842 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c1403e040 con 0x7f4c241549c0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c1affd640 1 -- 172.21.15.161:0/3872616842 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c14039cb0 con 0x7f4c241549c0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c140094d0 con 0x7f4c241549c0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c1affd640 1 -- 172.21.15.161:0/3872616842 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c14004270 con 0x7f4c241549c0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.066+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c24154f40 con 0x7f4c241549c0 2024-03-23T14:35:36.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.067+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c24154da0 con 0x7f4c241549c0 2024-03-23T14:35:36.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.067+0000 7f4c1affd640 1 -- 172.21.15.161:0/3872616842 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4c140043d0 con 0x7f4c241549c0 2024-03-23T14:35:36.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.068+0000 7f4c1affd640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4c0806b810 0x7f4c0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.068+0000 7f4c1affd640 1 -- 172.21.15.161:0/3872616842 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f4c140ac700 con 0x7f4c241549c0 2024-03-23T14:35:36.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.068+0000 7f4c2a407640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4c0806b810 0x7f4c0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.068+0000 7f4c2a407640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4c0806b810 0x7f4c0806dcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4c200012e0 tx=0x7f4c20009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.072+0000 7f4c1affd640 1 -- 172.21.15.161:0/3872616842 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4c24154da0 con 0x7f4c241549c0 2024-03-23T14:35:36.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.243+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4c24154da0 con 0x7f4c241549c0 2024-03-23T14:35:36.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.243+0000 7f4c1affd640 1 -- 172.21.15.161:0/3872616842 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f4c24154da0 con 0x7f4c241549c0 2024-03-23T14:35:36.239 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4c0806b810 msgr2=0x7f4c0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4c0806b810 0x7f4c0806dcd0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4c200012e0 tx=0x7f4c20009040 comp rx=0 tx=0).stop 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c241549c0 msgr2=0x7f4c24159230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c241549c0 0x7f4c24159230 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4c14009870 tx=0x7f4c14004730 comp rx=0 tx=0).stop 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 shutdown_connections 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4c0806b810 0x7f4c0806dcd0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4c2415b5b0 0x7f4c2414dec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4c241549c0 0x7f4c24159230 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 --2- 172.21.15.161:0/3872616842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4c2406e950 0x7f4c241f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 >> 172.21.15.161:0/3872616842 conn(0x7f4c2406b4a0 msgr2=0x7f4c2415c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:36.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 shutdown_connections 2024-03-23T14:35:36.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.245+0000 7f4c2c66b640 1 -- 172.21.15.161:0/3872616842 wait complete. 2024-03-23T14:35:36.251 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:36.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.360+0000 7f3547502640 1 Processor -- start 2024-03-23T14:35:36.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.360+0000 7f3547502640 1 -- start start 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3547502640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f35400b2aa0 0x7f35400cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3547502640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f35400d0600 con 0x7f35401552f0 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3547502640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f354015f480 con 0x7f35400d01b0 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3547502640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f35401556d0 con 0x7f35400b2aa0 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3544a9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f35400b2aa0 0x7f35400cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f354529e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f35401552f0 0x7f35400b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47934/0 (socket says 172.21.15.161:47934) 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f354529e640 1 -- 172.21.15.161:0/2080269740 learned_addr learned my addr 172.21.15.161:0/2080269740 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3544a9d640 1 -- 172.21.15.161:0/2080269740 >> v1:172.21.15.183:6789/0 conn(0x7f35400d01b0 legacy=0x7f354006ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3544a9d640 1 -- 172.21.15.161:0/2080269740 >> v1:172.21.15.161:6789/0 conn(0x7f35401552f0 legacy=0x7f35400b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3537fff640 1 -- 172.21.15.161:0/2080269740 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4281814802 0 0) 0x7f353c001110 con 0x7f35401552f0 2024-03-23T14:35:36.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.361+0000 7f3544a9d640 1 -- 172.21.15.161:0/2080269740 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f354015e870 con 0x7f35400b2aa0 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.362+0000 7f3544a9d640 1 --2- 172.21.15.161:0/2080269740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35400b2aa0 0x7f35400cfc70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f35300097a0 tx=0x7f353002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd5b735ba87a8e15 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.362+0000 7f3537fff640 1 -- 172.21.15.161:0/2080269740 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530030610 con 0x7f35400b2aa0 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.362+0000 7f3537fff640 1 -- 172.21.15.161:0/2080269740 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3530030770 con 0x7f35400b2aa0 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.362+0000 7f3537fff640 1 -- 172.21.15.161:0/2080269740 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3530030b00 con 0x7f35400b2aa0 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.362+0000 7f3547502640 1 -- 172.21.15.161:0/2080269740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35400b2aa0 msgr2=0x7f35400cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.362+0000 7f3547502640 1 --2- 172.21.15.161:0/2080269740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35400b2aa0 0x7f35400cfc70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f35300097a0 tx=0x7f353002b960 comp rx=0 tx=0).stop 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.363+0000 7f3547502640 1 -- 172.21.15.161:0/2080269740 shutdown_connections 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.363+0000 7f3547502640 1 --2- 172.21.15.161:0/2080269740 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35400b2aa0 0x7f35400cfc70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.363+0000 7f3547502640 1 -- 172.21.15.161:0/2080269740 >> 172.21.15.161:0/2080269740 conn(0x7f354006b260 msgr2=0x7f354006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.363+0000 7f3547502640 1 -- 172.21.15.161:0/2080269740 shutdown_connections 2024-03-23T14:35:36.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.363+0000 7f3547502640 1 -- 172.21.15.161:0/2080269740 wait complete. 2024-03-23T14:35:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.364+0000 7f3547502640 1 Processor -- start 2024-03-23T14:35:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.364+0000 7f3547502640 1 -- start start 2024-03-23T14:35:36.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.364+0000 7f3547502640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35400b2aa0 0x7f3540143e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3547502640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f35400d01b0 0x7f3540070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3547502640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35401552f0 0x7f35400710b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3547502640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3540144360 con 0x7f35401552f0 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3547502640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f35400cce60 con 0x7f35400b2aa0 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3547502640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f35400cd140 con 0x7f35400d01b0 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35401552f0 0x7f35400710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35401552f0 0x7f35400710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49700/0 (socket says 172.21.15.161:49700) 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 -- 172.21.15.161:0/1217201231 learned_addr learned my addr 172.21.15.161:0/1217201231 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f354529e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35400b2aa0 0x7f3540143e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3544a9d640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f35400d01b0 0x7f3540070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 -- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f35400d01b0 msgr2=0x7f3540070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f35400d01b0 0x7f3540070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 -- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35400b2aa0 msgr2=0x7f3540143e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35400b2aa0 0x7f3540143e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 -- 172.21.15.161:0/1217201231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3530009400 con 0x7f35401552f0 2024-03-23T14:35:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3545a9f640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35401552f0 0x7f35400710b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f353800c7a0 tx=0x7f353800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3535ffb640 1 -- 172.21.15.161:0/1217201231 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538003b70 con 0x7f35401552f0 2024-03-23T14:35:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3535ffb640 1 -- 172.21.15.161:0/1217201231 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3538003cf0 con 0x7f35401552f0 2024-03-23T14:35:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3535ffb640 1 -- 172.21.15.161:0/1217201231 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3538003ea0 con 0x7f35401552f0 2024-03-23T14:35:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35400cd420 con 0x7f35401552f0 2024-03-23T14:35:36.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.365+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35400cd9e0 con 0x7f35401552f0 2024-03-23T14:35:36.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.367+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f354004f4e0 con 0x7f35401552f0 2024-03-23T14:35:36.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.367+0000 7f3535ffb640 1 -- 172.21.15.161:0/1217201231 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f353800aa90 con 0x7f35401552f0 2024-03-23T14:35:36.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.367+0000 7f3535ffb640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f352406b810 0x7f352406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.367+0000 7f3535ffb640 1 -- 172.21.15.161:0/1217201231 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3538089ec0 con 0x7f35401552f0 2024-03-23T14:35:36.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.367+0000 7f354529e640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f352406b810 0x7f352406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.368+0000 7f354529e640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f352406b810 0x7f352406dcd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f353c000f60 tx=0x7f353c007e40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.371+0000 7f3535ffb640 1 -- 172.21.15.161:0/1217201231 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3538054990 con 0x7f35401552f0 2024-03-23T14:35:36.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.544+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f354015ad90 con 0x7f35401552f0 2024-03-23T14:35:36.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.544+0000 7f3535ffb640 1 -- 172.21.15.161:0/1217201231 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f3538058050 con 0x7f35401552f0 2024-03-23T14:35:36.540 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:36.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f352406b810 msgr2=0x7f352406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f352406b810 0x7f352406dcd0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f353c000f60 tx=0x7f353c007e40 comp rx=0 tx=0).stop 2024-03-23T14:35:36.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35401552f0 msgr2=0x7f35400710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35401552f0 0x7f35400710b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f353800c7a0 tx=0x7f353800cc60 comp rx=0 tx=0).stop 2024-03-23T14:35:36.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 shutdown_connections 2024-03-23T14:35:36.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f352406b810 0x7f352406dcd0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f35401552f0 0x7f35400710b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f35400d01b0 0x7f3540070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 --2- 172.21.15.161:0/1217201231 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f35400b2aa0 0x7f3540143e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 >> 172.21.15.161:0/1217201231 conn(0x7f354006b260 msgr2=0x7f35400af670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:36.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 shutdown_connections 2024-03-23T14:35:36.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.546+0000 7f3547502640 1 -- 172.21.15.161:0/1217201231 wait complete. 2024-03-23T14:35:36.553 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:36.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.656+0000 7f5d5035e640 1 Processor -- start 2024-03-23T14:35:36.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.656+0000 7f5d5035e640 1 -- start start 2024-03-23T14:35:36.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d5035e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5d4815e400 0x7f5d480cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d5035e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5d480d2b40 0x7f5d480b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d5035e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d480d0460 con 0x7f5d480cf9d0 2024-03-23T14:35:36.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d5035e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5d480d1050 con 0x7f5d480d2b40 2024-03-23T14:35:36.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d5035e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f5d480d06b0 con 0x7f5d4815e400 2024-03-23T14:35:36.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d4e8fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5d480d2b40 0x7f5d480b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d4d8f9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5d480cf9d0 0x7f5d480d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47950/0 (socket says 172.21.15.161:47950) 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d4d8f9640 1 -- 172.21.15.161:0/2325564272 learned_addr learned my addr 172.21.15.161:0/2325564272 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.657+0000 7f5d4e0fa640 1 --2- 172.21.15.161:0/2325564272 >> v2:172.21.15.183:3300/0 conn(0x7f5d4815e400 0x7f5d480cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.658+0000 7f5d4e8fb640 1 -- 172.21.15.161:0/2325564272 >> v2:172.21.15.183:3300/0 conn(0x7f5d4815e400 msgr2=0x7f5d480cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.658+0000 7f5d4e8fb640 1 --2- 172.21.15.161:0/2325564272 >> v2:172.21.15.183:3300/0 conn(0x7f5d4815e400 0x7f5d480cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.658+0000 7f5d4e8fb640 1 -- 172.21.15.161:0/2325564272 >> v1:172.21.15.161:6789/0 conn(0x7f5d480cf9d0 legacy=0x7f5d480d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.658+0000 7f5d4e8fb640 1 -- 172.21.15.161:0/2325564272 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d480d0990 con 0x7f5d480d2b40 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.658+0000 7f5d4d0f8640 1 -- 172.21.15.161:0/2325564272 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2474953159 0 0) 0x7f5d480d0460 con 0x7f5d480cf9d0 2024-03-23T14:35:36.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.658+0000 7f5d4e0fa640 1 --2- 172.21.15.161:0/2325564272 >> v2:172.21.15.183:3300/0 conn(0x7f5d4815e400 0x7f5d480cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:36.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.658+0000 7f5d4e8fb640 1 --2- 172.21.15.161:0/2325564272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d480b0a80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5d4808e230 tx=0x7f5d4002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845ba8ffa9fbebaa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d4d0f8640 1 -- 172.21.15.161:0/2325564272 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40031830 con 0x7f5d480d2b40 2024-03-23T14:35:36.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d4d0f8640 1 -- 172.21.15.161:0/2325564272 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d40031990 con 0x7f5d480d2b40 2024-03-23T14:35:36.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d4d0f8640 1 -- 172.21.15.161:0/2325564272 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d40031d20 con 0x7f5d480d2b40 2024-03-23T14:35:36.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d5035e640 1 -- 172.21.15.161:0/2325564272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 msgr2=0x7f5d480b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d5035e640 1 --2- 172.21.15.161:0/2325564272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d480b0a80 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5d4808e230 tx=0x7f5d4002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d5035e640 1 -- 172.21.15.161:0/2325564272 shutdown_connections 2024-03-23T14:35:36.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d5035e640 1 --2- 172.21.15.161:0/2325564272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d480b0a80 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d5035e640 1 --2- 172.21.15.161:0/2325564272 >> v2:172.21.15.183:3300/0 conn(0x7f5d4815e400 0x7f5d480cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d5035e640 1 -- 172.21.15.161:0/2325564272 >> 172.21.15.161:0/2325564272 conn(0x7f5d4809d190 msgr2=0x7f5d480d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d5035e640 1 -- 172.21.15.161:0/2325564272 shutdown_connections 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.659+0000 7f5d5035e640 1 -- 172.21.15.161:0/2325564272 wait complete. 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.660+0000 7f5d5035e640 1 Processor -- start 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.660+0000 7f5d5035e640 1 -- start start 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d5035e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d480cf9d0 0x7f5d481566c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d5035e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d48156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d5035e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4815e400 0x7f5d480cb100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d5035e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48157570 con 0x7f5d480d2b40 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d5035e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5d48157850 con 0x7f5d480cf9d0 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d5035e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5d48157b30 con 0x7f5d4815e400 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4d8f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d48156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4d8f9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d48156ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49720/0 (socket says 172.21.15.161:49720) 2024-03-23T14:35:36.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4d8f9640 1 -- 172.21.15.161:0/1113700586 learned_addr learned my addr 172.21.15.161:0/1113700586 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4e0fa640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d480cf9d0 0x7f5d481566c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4e8fb640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4815e400 0x7f5d480cb100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4d8f9640 1 -- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4815e400 msgr2=0x7f5d480cb100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4d8f9640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4815e400 0x7f5d480cb100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4d8f9640 1 -- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d480cf9d0 msgr2=0x7f5d481566c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4d8f9640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d480cf9d0 0x7f5d481566c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.661+0000 7f5d4d8f9640 1 -- 172.21.15.161:0/1113700586 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d44009a70 con 0x7f5d480d2b40 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.662+0000 7f5d4d8f9640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d48156ce0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800c4a0 tx=0x7f5d3800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.662+0000 7f5d3effd640 1 -- 172.21.15.161:0/1113700586 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d380074e0 con 0x7f5d480d2b40 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.662+0000 7f5d3effd640 1 -- 172.21.15.161:0/1113700586 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d38007640 con 0x7f5d480d2b40 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.662+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d40009400 con 0x7f5d480d2b40 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.662+0000 7f5d3effd640 1 -- 172.21.15.161:0/1113700586 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d38004a00 con 0x7f5d480d2b40 2024-03-23T14:35:36.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.662+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d481580f0 con 0x7f5d480d2b40 2024-03-23T14:35:36.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.663+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d48158310 con 0x7f5d480d2b40 2024-03-23T14:35:36.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.663+0000 7f5d3effd640 1 -- 172.21.15.161:0/1113700586 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5d38004b60 con 0x7f5d480d2b40 2024-03-23T14:35:36.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.664+0000 7f5d3effd640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d2c06b810 0x7f5d2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.664+0000 7f5d3effd640 1 -- 172.21.15.161:0/1113700586 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f5d380895c0 con 0x7f5d480d2b40 2024-03-23T14:35:36.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.664+0000 7f5d4e0fa640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d2c06b810 0x7f5d2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.664+0000 7f5d4e0fa640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d2c06b810 0x7f5d2c06dcd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5d440114e0 tx=0x7f5d4400b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.668+0000 7f5d3effd640 1 -- 172.21.15.161:0/1113700586 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5d38054120 con 0x7f5d480d2b40 2024-03-23T14:35:36.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.843+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d480a1e00 con 0x7f5d480d2b40 2024-03-23T14:35:36.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.844+0000 7f5d3effd640 1 -- 172.21.15.161:0/1113700586 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f5d380577e0 con 0x7f5d480d2b40 2024-03-23T14:35:36.840 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:36.841 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d2c06b810 msgr2=0x7f5d2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d2c06b810 0x7f5d2c06dcd0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f5d440114e0 tx=0x7f5d4400b040 comp rx=0 tx=0).stop 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 msgr2=0x7f5d48156ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d48156ce0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f5d3800c4a0 tx=0x7f5d3800c960 comp rx=0 tx=0).stop 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 shutdown_connections 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d2c06b810 0x7f5d2c06dcd0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d4815e400 0x7f5d480cb100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d480d2b40 0x7f5d48156ce0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 --2- 172.21.15.161:0/1113700586 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d480cf9d0 0x7f5d481566c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 >> 172.21.15.161:0/1113700586 conn(0x7f5d4809d190 msgr2=0x7f5d48162620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:36.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 shutdown_connections 2024-03-23T14:35:36.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.847+0000 7f5d5035e640 1 -- 172.21.15.161:0/1113700586 wait complete. 2024-03-23T14:35:36.853 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:36.854 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.962+0000 7f509aba0640 1 Processor -- start 2024-03-23T14:35:36.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.962+0000 7f509aba0640 1 -- start start 2024-03-23T14:35:36.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f509aba0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f50940cc3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f509aba0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f509415ef20 0x7f50940cdea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f509aba0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50940b3420 con 0x7f509415f3b0 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f509aba0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f5094144770 con 0x7f509415ef20 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f509aba0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f50940cc930 con 0x7f5094144390 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f509939d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f509415f3b0 0x7f509415e810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47958/0 (socket says 172.21.15.161:47958) 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f509939d640 1 -- 172.21.15.161:0/1998459997 learned_addr learned my addr 172.21.15.161:0/1998459997 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f5099b9e640 1 --2- 172.21.15.161:0/1998459997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f50940cc3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f509a39f640 1 --2- 172.21.15.161:0/1998459997 >> v2:172.21.15.183:3300/0 conn(0x7f509415ef20 0x7f50940cdea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.963+0000 7f5098b9c640 1 -- 172.21.15.161:0/1998459997 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1297816110 0 0) 0x7f50940b3420 con 0x7f509415f3b0 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.964+0000 7f5098b9c640 1 -- 172.21.15.161:0/1998459997 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5080003520 con 0x7f509415f3b0 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.964+0000 7f5099b9e640 1 -- 172.21.15.161:0/1998459997 >> v2:172.21.15.183:3300/0 conn(0x7f509415ef20 msgr2=0x7f50940cdea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.964+0000 7f5099b9e640 1 --2- 172.21.15.161:0/1998459997 >> v2:172.21.15.183:3300/0 conn(0x7f509415ef20 0x7f50940cdea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.964+0000 7f5099b9e640 1 -- 172.21.15.161:0/1998459997 >> v1:172.21.15.161:6789/0 conn(0x7f509415f3b0 legacy=0x7f509415e810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.964+0000 7f5099b9e640 1 -- 172.21.15.161:0/1998459997 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50940b4ab0 con 0x7f5094144390 2024-03-23T14:35:36.961 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.964+0000 7f509a39f640 1 --2- 172.21.15.161:0/1998459997 >> v2:172.21.15.183:3300/0 conn(0x7f509415ef20 0x7f50940cdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.964+0000 7f5099b9e640 1 --2- 172.21.15.161:0/1998459997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f50940cc3f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5090009d90 tx=0x7f509002bd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3949c6104fae7055 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f5098b9c640 1 -- 172.21.15.161:0/1998459997 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f509002cb30 con 0x7f5094144390 2024-03-23T14:35:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f5098b9c640 1 -- 172.21.15.161:0/1998459997 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f509002cc90 con 0x7f5094144390 2024-03-23T14:35:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f5098b9c640 1 -- 172.21.15.161:0/1998459997 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5090034440 con 0x7f5094144390 2024-03-23T14:35:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f509aba0640 1 -- 172.21.15.161:0/1998459997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 msgr2=0x7f50940cc3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f509aba0640 1 --2- 172.21.15.161:0/1998459997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f50940cc3f0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f5090009d90 tx=0x7f509002bd80 comp rx=0 tx=0).stop 2024-03-23T14:35:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f509aba0640 1 -- 172.21.15.161:0/1998459997 shutdown_connections 2024-03-23T14:35:36.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f509aba0640 1 --2- 172.21.15.161:0/1998459997 >> v2:172.21.15.183:3300/0 conn(0x7f509415ef20 0x7f50940cdea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f509aba0640 1 --2- 172.21.15.161:0/1998459997 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f50940cc3f0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f509aba0640 1 -- 172.21.15.161:0/1998459997 >> 172.21.15.161:0/1998459997 conn(0x7f509406b260 msgr2=0x7f509406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f509aba0640 1 -- 172.21.15.161:0/1998459997 shutdown_connections 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.965+0000 7f509aba0640 1 -- 172.21.15.161:0/1998459997 wait complete. 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.966+0000 7f509aba0640 1 Processor -- start 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.966+0000 7f509aba0640 1 -- start start 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509aba0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f509414c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509aba0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f509415ef20 0x7f509413d590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509aba0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509415f3b0 0x7f509415d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509aba0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f509413ba80 con 0x7f509415f3b0 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509aba0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f509413bd60 con 0x7f509415ef20 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509aba0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f509413c040 con 0x7f5094144390 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509a39f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509415f3b0 0x7f509415d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509a39f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509415f3b0 0x7f509415d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49726/0 (socket says 172.21.15.161:49726) 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509a39f640 1 -- 172.21.15.161:0/1417885040 learned_addr learned my addr 172.21.15.161:0/1417885040 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f5099b9e640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f509414c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.967+0000 7f509939d640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f509415ef20 0x7f509413d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f509a39f640 1 -- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 msgr2=0x7f509414c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f509a39f640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f509414c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f509a39f640 1 -- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f509415ef20 msgr2=0x7f509413d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f509a39f640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f509415ef20 0x7f509413d590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f509a39f640 1 -- 172.21.15.161:0/1417885040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f508c009400 con 0x7f509415f3b0 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f509a39f640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509415f3b0 0x7f509415d040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f508c00e930 tx=0x7f508c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f508a7fc640 1 -- 172.21.15.161:0/1417885040 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c009700 con 0x7f509415f3b0 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f508a7fc640 1 -- 172.21.15.161:0/1417885040 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f508c00ca00 con 0x7f509415f3b0 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f508a7fc640 1 -- 172.21.15.161:0/1417885040 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f508c00cd90 con 0x7f509415f3b0 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5090009a70 con 0x7f509415f3b0 2024-03-23T14:35:36.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.968+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f509413c600 con 0x7f509415f3b0 2024-03-23T14:35:36.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.969+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f509413c7f0 con 0x7f509415f3b0 2024-03-23T14:35:36.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.969+0000 7f508a7fc640 1 -- 172.21.15.161:0/1417885040 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f508c013470 con 0x7f509415f3b0 2024-03-23T14:35:36.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.970+0000 7f508a7fc640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f507806b840 0x7f507806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:36.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.970+0000 7f508a7fc640 1 -- 172.21.15.161:0/1417885040 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f508c089d90 con 0x7f509415f3b0 2024-03-23T14:35:36.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.970+0000 7f5099b9e640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f507806b840 0x7f507806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:36.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.970+0000 7f5099b9e640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f507806b840 0x7f507806dd00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5090036000 tx=0x7f5090030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:36.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:36.974+0000 7f508a7fc640 1 -- 172.21.15.161:0/1417885040 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f508c0548a0 con 0x7f509415f3b0 2024-03-23T14:35:36.996 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:35:36.998 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:35:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.150+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f509404f470 con 0x7f509415f3b0 2024-03-23T14:35:37.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.150+0000 7f508a7fc640 1 -- 172.21.15.161:0/1417885040 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f508c057f60 con 0x7f509415f3b0 2024-03-23T14:35:37.146 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f507806b840 msgr2=0x7f507806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f507806b840 0x7f507806dd00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5090036000 tx=0x7f5090030000 comp rx=0 tx=0).stop 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509415f3b0 msgr2=0x7f509415d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509415f3b0 0x7f509415d040 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f508c00e930 tx=0x7f508c00edf0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 shutdown_connections 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f507806b840 0x7f507806dd00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f509415f3b0 0x7f509415d040 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f509415ef20 0x7f509413d590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 --2- 172.21.15.161:0/1417885040 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5094144390 0x7f509414c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.149 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 >> 172.21.15.161:0/1417885040 conn(0x7f509406b260 msgr2=0x7f50940b09f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:37.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 shutdown_connections 2024-03-23T14:35:37.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.153+0000 7f509aba0640 1 -- 172.21.15.161:0/1417885040 wait complete. 2024-03-23T14:35:37.160 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:37.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.269+0000 7fbe7a00b640 1 Processor -- start 2024-03-23T14:35:37.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.269+0000 7fbe7a00b640 1 -- start start 2024-03-23T14:35:37.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe7a00b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbe7415e1d0 0x7fbe740b4570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe7a00b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbe740b3c10 0x7fbe740d02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe7a00b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe740b2c20 con 0x7fbe740cfee0 2024-03-23T14:35:37.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe7a00b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbe740b37f0 con 0x7fbe7415e1d0 2024-03-23T14:35:37.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe7a00b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fbe740b4060 con 0x7fbe740b3c10 2024-03-23T14:35:37.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe72ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbe7415e1d0 0x7fbe740b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe737fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbe740cfee0 0x7fbe7406d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47960/0 (socket says 172.21.15.161:47960) 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe737fe640 1 -- 172.21.15.161:0/2749094161 learned_addr learned my addr 172.21.15.161:0/2749094161 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.270+0000 7fbe73fff640 1 --2- 172.21.15.161:0/2749094161 >> v2:172.21.15.183:3300/0 conn(0x7fbe740b3c10 0x7fbe740d02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.271+0000 7fbe72ffd640 1 -- 172.21.15.161:0/2749094161 >> v2:172.21.15.183:3300/0 conn(0x7fbe740b3c10 msgr2=0x7fbe740d02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.271+0000 7fbe72ffd640 1 --2- 172.21.15.161:0/2749094161 >> v2:172.21.15.183:3300/0 conn(0x7fbe740b3c10 0x7fbe740d02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.271+0000 7fbe72ffd640 1 -- 172.21.15.161:0/2749094161 >> v1:172.21.15.161:6789/0 conn(0x7fbe740cfee0 legacy=0x7fbe7406d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.271+0000 7fbe727fc640 1 -- 172.21.15.161:0/2749094161 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311690530 0 0) 0x7fbe740b2c20 con 0x7fbe740cfee0 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.271+0000 7fbe72ffd640 1 -- 172.21.15.161:0/2749094161 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe740d0800 con 0x7fbe7415e1d0 2024-03-23T14:35:37.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.271+0000 7fbe73fff640 1 --2- 172.21.15.161:0/2749094161 >> v2:172.21.15.183:3300/0 conn(0x7fbe740b3c10 0x7fbe740d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:37.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.272+0000 7fbe72ffd640 1 --2- 172.21.15.161:0/2749094161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe7415e1d0 0x7fbe740b4570 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbe740aec30 tx=0x7fbe6402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db5e886fc92831e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.272+0000 7fbe727fc640 1 -- 172.21.15.161:0/2749094161 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64030610 con 0x7fbe7415e1d0 2024-03-23T14:35:37.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.272+0000 7fbe727fc640 1 -- 172.21.15.161:0/2749094161 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe64030770 con 0x7fbe7415e1d0 2024-03-23T14:35:37.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.272+0000 7fbe727fc640 1 -- 172.21.15.161:0/2749094161 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe64030b00 con 0x7fbe7415e1d0 2024-03-23T14:35:37.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.273+0000 7fbe7a00b640 1 -- 172.21.15.161:0/2749094161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe7415e1d0 msgr2=0x7fbe740b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.273+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/2749094161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe7415e1d0 0x7fbe740b4570 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fbe740aec30 tx=0x7fbe6402b960 comp rx=0 tx=0).stop 2024-03-23T14:35:37.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.273+0000 7fbe7a00b640 1 -- 172.21.15.161:0/2749094161 shutdown_connections 2024-03-23T14:35:37.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.273+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/2749094161 >> v2:172.21.15.183:3300/0 conn(0x7fbe740b3c10 0x7fbe740d02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.273+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/2749094161 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe7415e1d0 0x7fbe740b4570 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.273+0000 7fbe7a00b640 1 -- 172.21.15.161:0/2749094161 >> 172.21.15.161:0/2749094161 conn(0x7fbe7406b4a0 msgr2=0x7fbe7406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:37.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.273+0000 7fbe7a00b640 1 -- 172.21.15.161:0/2749094161 shutdown_connections 2024-03-23T14:35:37.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.273+0000 7fbe7a00b640 1 -- 172.21.15.161:0/2749094161 wait complete. 2024-03-23T14:35:37.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.274+0000 7fbe7a00b640 1 Processor -- start 2024-03-23T14:35:37.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.274+0000 7fbe7a00b640 1 -- start start 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe7a00b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe740b3c10 0x7fbe7414b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe7a00b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe740cfee0 0x7fbe741f3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe7a00b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe7415e1d0 0x7fbe740ce3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe7a00b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7415ef80 con 0x7fbe740b3c10 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe7a00b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbe7415fac0 con 0x7fbe740cfee0 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe7a00b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fbe740cd7a0 con 0x7fbe7415e1d0 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe740b3c10 0x7fbe7414b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe740b3c10 0x7fbe7414b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49736/0 (socket says 172.21.15.161:49736) 2024-03-23T14:35:37.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 -- 172.21.15.161:0/3610898293 learned_addr learned my addr 172.21.15.161:0/3610898293 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe72ffd640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe740cfee0 0x7fbe741f3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe73fff640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe7415e1d0 0x7fbe740ce3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 -- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe7415e1d0 msgr2=0x7fbe740ce3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe7415e1d0 0x7fbe740ce3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 -- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe740cfee0 msgr2=0x7fbe741f3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe740cfee0 0x7fbe741f3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 -- 172.21.15.161:0/3610898293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6c009a70 con 0x7fbe740b3c10 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.275+0000 7fbe737fe640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe740b3c10 0x7fbe7414b530 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800c500 tx=0x7fbe6800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.276+0000 7fbe5bfff640 1 -- 172.21.15.161:0/3610898293 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe680074e0 con 0x7fbe740b3c10 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.276+0000 7fbe5bfff640 1 -- 172.21.15.161:0/3610898293 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbe68007660 con 0x7fbe740b3c10 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.276+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe64009400 con 0x7fbe740b3c10 2024-03-23T14:35:37.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.276+0000 7fbe5bfff640 1 -- 172.21.15.161:0/3610898293 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe68004a20 con 0x7fbe740b3c10 2024-03-23T14:35:37.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.276+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe7415f470 con 0x7fbe740b3c10 2024-03-23T14:35:37.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.277+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe74073570 con 0x7fbe740b3c10 2024-03-23T14:35:37.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.277+0000 7fbe5bfff640 1 -- 172.21.15.161:0/3610898293 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbe68004b80 con 0x7fbe740b3c10 2024-03-23T14:35:37.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.277+0000 7fbe5bfff640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbe6006b810 0x7fbe6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.278+0000 7fbe5bfff640 1 -- 172.21.15.161:0/3610898293 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fbe68089380 con 0x7fbe740b3c10 2024-03-23T14:35:37.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.278+0000 7fbe72ffd640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbe6006b810 0x7fbe6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.278+0000 7fbe72ffd640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbe6006b810 0x7fbe6006dcd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbe64036000 tx=0x7fbe6403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.282+0000 7fbe5bfff640 1 -- 172.21.15.161:0/3610898293 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbe68053d40 con 0x7fbe740b3c10 2024-03-23T14:35:37.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.456+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fbe7404f550 con 0x7fbe740b3c10 2024-03-23T14:35:37.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.457+0000 7fbe5bfff640 1 -- 172.21.15.161:0/3610898293 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fbe68057400 con 0x7fbe740b3c10 2024-03-23T14:35:37.453 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbe6006b810 msgr2=0x7fbe6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbe6006b810 0x7fbe6006dcd0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fbe64036000 tx=0x7fbe6403b040 comp rx=0 tx=0).stop 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe740b3c10 msgr2=0x7fbe7414b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe740b3c10 0x7fbe7414b530 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fbe6800c500 tx=0x7fbe6800c900 comp rx=0 tx=0).stop 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 shutdown_connections 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbe6006b810 0x7fbe6006dcd0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbe7415e1d0 0x7fbe740ce3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbe740cfee0 0x7fbe741f3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 --2- 172.21.15.161:0/3610898293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbe740b3c10 0x7fbe7414b530 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 >> 172.21.15.161:0/3610898293 conn(0x7fbe7406b4a0 msgr2=0x7fbe740af680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:37.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 shutdown_connections 2024-03-23T14:35:37.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.459+0000 7fbe7a00b640 1 -- 172.21.15.161:0/3610898293 wait complete. 2024-03-23T14:35:37.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:37.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.578+0000 7f58a5e55640 1 Processor -- start 2024-03-23T14:35:37.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.578+0000 7f58a5e55640 1 -- start start 2024-03-23T14:35:37.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f58a5e55640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f58a00d1270 0x7f58a00d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f58a5e55640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58a0161c00 con 0x7f58a00d29d0 2024-03-23T14:35:37.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f58a5e55640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58a00d2e20 con 0x7f58a0161820 2024-03-23T14:35:37.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f58a5e55640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f58a00cea30 con 0x7f58a00d1270 2024-03-23T14:35:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f589ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f58a00d29d0 0x7f58a00b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47974/0 (socket says 172.21.15.161:47974) 2024-03-23T14:35:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f589ffff640 1 -- 172.21.15.161:0/2016891866 learned_addr learned my addr 172.21.15.161:0/2016891866 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f589effd640 1 --2- 172.21.15.161:0/2016891866 >> v2:172.21.15.183:3300/0 conn(0x7f58a00d1270 0x7f58a00d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f589e7fc640 1 -- 172.21.15.161:0/2016891866 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1984090869 0 0) 0x7f58a0161c00 con 0x7f58a00d29d0 2024-03-23T14:35:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f589e7fc640 1 -- 172.21.15.161:0/2016891866 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f588c003520 con 0x7f58a00d29d0 2024-03-23T14:35:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f589effd640 1 -- 172.21.15.161:0/2016891866 >> v1:172.21.15.183:6789/0 conn(0x7f58a0161820 legacy=0x7f58a00d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.579+0000 7f589e7fc640 1 -- 172.21.15.161:0/2016891866 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 114263240 0 0) 0x7f588c003520 con 0x7f58a00d29d0 2024-03-23T14:35:37.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.580+0000 7f589effd640 1 -- 172.21.15.161:0/2016891866 >> v1:172.21.15.161:6789/0 conn(0x7f58a00d29d0 legacy=0x7f58a00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.580+0000 7f589effd640 1 -- 172.21.15.161:0/2016891866 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58a00cfd70 con 0x7f58a00d1270 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.580+0000 7f589effd640 1 --2- 172.21.15.161:0/2016891866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 0x7f58a00d2490 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f58880097a0 tx=0x7f588802b680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6729ddcebe9c7b41 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.580+0000 7f589e7fc640 1 -- 172.21.15.161:0/2016891866 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5890001670 con 0x7f58a00d1270 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.580+0000 7f589e7fc640 1 -- 172.21.15.161:0/2016891866 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f588802e050 con 0x7f58a00d1270 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.580+0000 7f589e7fc640 1 -- 172.21.15.161:0/2016891866 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5888030730 con 0x7f58a00d1270 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.581+0000 7f58a5e55640 1 -- 172.21.15.161:0/2016891866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 msgr2=0x7f58a00d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.581+0000 7f58a5e55640 1 --2- 172.21.15.161:0/2016891866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 0x7f58a00d2490 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f58880097a0 tx=0x7f588802b680 comp rx=0 tx=0).stop 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.581+0000 7f58a5e55640 1 -- 172.21.15.161:0/2016891866 shutdown_connections 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.581+0000 7f58a5e55640 1 --2- 172.21.15.161:0/2016891866 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 0x7f58a00d2490 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.581+0000 7f58a5e55640 1 -- 172.21.15.161:0/2016891866 >> 172.21.15.161:0/2016891866 conn(0x7f58a009d190 msgr2=0x7f58a00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.581+0000 7f58a5e55640 1 -- 172.21.15.161:0/2016891866 shutdown_connections 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.581+0000 7f58a5e55640 1 -- 172.21.15.161:0/2016891866 wait complete. 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.583+0000 7f58a5e55640 1 Processor -- start 2024-03-23T14:35:37.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.583+0000 7f58a5e55640 1 -- start start 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.583+0000 7f58a5e55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 0x7f58a00caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.583+0000 7f58a5e55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58a00d29d0 0x7f58a00ca490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.583+0000 7f58a5e55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a0161820 0x7f58a00cda30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.583+0000 7f58a5e55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f58a00cd610 con 0x7f58a0161820 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.583+0000 7f58a5e55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f58a00cb560 con 0x7f58a00d1270 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.583+0000 7f58a5e55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f58a00cb840 con 0x7f58a00d29d0 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a0161820 0x7f58a00cda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a0161820 0x7f58a00cda30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49752/0 (socket says 172.21.15.161:49752) 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 -- 172.21.15.161:0/1423775811 learned_addr learned my addr 172.21.15.161:0/1423775811 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589f7fe640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 0x7f58a00caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589effd640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58a00d29d0 0x7f58a00ca490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 -- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58a00d29d0 msgr2=0x7f58a00ca490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58a00d29d0 0x7f58a00ca490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 -- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 msgr2=0x7f58a00caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 0x7f58a00caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 -- 172.21.15.161:0/1423775811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5888009400 con 0x7f58a0161820 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f589ffff640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a0161820 0x7f58a00cda30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f589000c750 tx=0x7f589000cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f5897fff640 1 -- 172.21.15.161:0/1423775811 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58900072d0 con 0x7f58a0161820 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f5897fff640 1 -- 172.21.15.161:0/1423775811 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5890007430 con 0x7f58a0161820 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58a00cbb20 con 0x7f58a0161820 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f5897fff640 1 -- 172.21.15.161:0/1423775811 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58900047e0 con 0x7f58a0161820 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.584+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58a015a9b0 con 0x7f58a0161820 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.585+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58a00cd750 con 0x7f58a0161820 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.586+0000 7f5897fff640 1 -- 172.21.15.161:0/1423775811 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5890004940 con 0x7f58a0161820 2024-03-23T14:35:37.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.586+0000 7f5897fff640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f588006b810 0x7f588006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.586+0000 7f5897fff640 1 -- 172.21.15.161:0/1423775811 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f58900896f0 con 0x7f58a0161820 2024-03-23T14:35:37.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.586+0000 7f589f7fe640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f588006b810 0x7f588006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.587+0000 7f589f7fe640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f588006b810 0x7f588006dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f58980012e0 tx=0x7f5898009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.590+0000 7f5897fff640 1 -- 172.21.15.161:0/1423775811 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f58a00cd750 con 0x7f58a0161820 2024-03-23T14:35:37.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.769+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f58a00cd750 con 0x7f58a0161820 2024-03-23T14:35:37.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.769+0000 7f5897fff640 1 -- 172.21.15.161:0/1423775811 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f58a00cd750 con 0x7f58a0161820 2024-03-23T14:35:37.765 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:37.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f588006b810 msgr2=0x7f588006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f588006b810 0x7f588006dcd0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f58980012e0 tx=0x7f5898009040 comp rx=0 tx=0).stop 2024-03-23T14:35:37.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a0161820 msgr2=0x7f58a00cda30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a0161820 0x7f58a00cda30 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f589000c750 tx=0x7f589000cc10 comp rx=0 tx=0).stop 2024-03-23T14:35:37.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 shutdown_connections 2024-03-23T14:35:37.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f588006b810 0x7f588006dcd0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f58a0161820 0x7f58a00cda30 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f58a00d29d0 0x7f58a00ca490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 --2- 172.21.15.161:0/1423775811 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f58a00d1270 0x7f58a00caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.771+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 >> 172.21.15.161:0/1423775811 conn(0x7f58a009d190 msgr2=0x7f58a006b930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:37.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.772+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 shutdown_connections 2024-03-23T14:35:37.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.772+0000 7f58a5e55640 1 -- 172.21.15.161:0/1423775811 wait complete. 2024-03-23T14:35:37.805 INFO:tasks.cephfs_test_runner:Starting test: test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) That a client who does not have the capability for setting 2024-03-23T14:35:37.805 INFO:tasks.cephfs_test_runner:test_forbidden_modification (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-23T14:35:37.805 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:37.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.914+0000 7f0a50e07640 1 Processor -- start 2024-03-23T14:35:37.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.914+0000 7f0a50e07640 1 -- start start 2024-03-23T14:35:37.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a50e07640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a4c15e400 0x7f0a4c0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a50e07640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a4c0d0460 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a50e07640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a4c0d06b0 con 0x7f0a4c0cf9d0 2024-03-23T14:35:37.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a50e07640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0a4c0d1050 con 0x7f0a4c15e400 2024-03-23T14:35:37.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a4ad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0a4c0d2b40 0x7f0a4c0b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47986/0 (socket says 172.21.15.161:47986) 2024-03-23T14:35:37.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a4a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0a4c15e400 0x7f0a4c0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a4ad9d640 1 -- 172.21.15.161:0/1868256490 learned_addr learned my addr 172.21.15.161:0/1868256490 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:37.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a4a59c640 1 -- 172.21.15.161:0/1868256490 >> v1:172.21.15.183:6789/0 conn(0x7f0a4c0cf9d0 legacy=0x7f0a4c0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a4a59c640 1 -- 172.21.15.161:0/1868256490 >> v1:172.21.15.161:6789/0 conn(0x7f0a4c0d2b40 legacy=0x7f0a4c0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.915+0000 7f0a4a59c640 1 -- 172.21.15.161:0/1868256490 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a4c0d0990 con 0x7f0a4c15e400 2024-03-23T14:35:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.916+0000 7f0a4a59c640 1 --2- 172.21.15.161:0/1868256490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c15e400 0x7f0a4c0cf490 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c0097a0 tx=0x7f0a3c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a2de8766f3b2871 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.916+0000 7f0a49d9b640 1 -- 172.21.15.161:0/1868256490 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c030610 con 0x7f0a4c15e400 2024-03-23T14:35:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.916+0000 7f0a49d9b640 1 -- 172.21.15.161:0/1868256490 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a3c030770 con 0x7f0a4c15e400 2024-03-23T14:35:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.916+0000 7f0a49d9b640 1 -- 172.21.15.161:0/1868256490 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a3c030b00 con 0x7f0a4c15e400 2024-03-23T14:35:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.917+0000 7f0a50e07640 1 -- 172.21.15.161:0/1868256490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c15e400 msgr2=0x7f0a4c0cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.917+0000 7f0a50e07640 1 --2- 172.21.15.161:0/1868256490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c15e400 0x7f0a4c0cf490 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c0097a0 tx=0x7f0a3c02b960 comp rx=0 tx=0).stop 2024-03-23T14:35:37.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.917+0000 7f0a50e07640 1 -- 172.21.15.161:0/1868256490 shutdown_connections 2024-03-23T14:35:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.917+0000 7f0a50e07640 1 --2- 172.21.15.161:0/1868256490 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c15e400 0x7f0a4c0cf490 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.917+0000 7f0a50e07640 1 -- 172.21.15.161:0/1868256490 >> 172.21.15.161:0/1868256490 conn(0x7f0a4c09d190 msgr2=0x7f0a4c0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.917+0000 7f0a50e07640 1 -- 172.21.15.161:0/1868256490 shutdown_connections 2024-03-23T14:35:37.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.917+0000 7f0a50e07640 1 -- 172.21.15.161:0/1868256490 wait complete. 2024-03-23T14:35:37.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.918+0000 7f0a50e07640 1 Processor -- start 2024-03-23T14:35:37.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.918+0000 7f0a50e07640 1 -- start start 2024-03-23T14:35:37.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a50e07640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a4c0cf9d0 0x7f0a4c157100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a50e07640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c0d2b40 0x7f0a4c0c8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a50e07640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a4c15e400 0x7f0a4c0b8630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a50e07640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c0ca120 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a50e07640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0a4c0c8510 con 0x7f0a4c15e400 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a50e07640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0a4c0c87f0 con 0x7f0a4c0cf9d0 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c0d2b40 0x7f0a4c0c8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a4a59c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a4c0cf9d0 0x7f0a4c157100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a4ad9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a4c15e400 0x7f0a4c0b8630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a43fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c0d2b40 0x7f0a4c0c8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49774/0 (socket says 172.21.15.161:49774) 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a4a59c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a4c0cf9d0 0x7f0a4c157100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:47424/0 (socket says 172.21.15.161:47424) 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.919+0000 7f0a43fff640 1 -- 172.21.15.161:0/3298062321 learned_addr learned my addr 172.21.15.161:0/3298062321 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a43fff640 1 -- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a4c0cf9d0 msgr2=0x7f0a4c157100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a43fff640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a4c0cf9d0 0x7f0a4c157100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a43fff640 1 -- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a4c15e400 msgr2=0x7f0a4c0b8630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a43fff640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a4c15e400 0x7f0a4c0b8630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a43fff640 1 -- 172.21.15.161:0/3298062321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a3c009400 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a4ad9d640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a4c15e400 0x7f0a4c0b8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:37.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a43fff640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c0d2b40 0x7f0a4c0c8ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400c400 tx=0x7f0a3400c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a42ffd640 1 -- 172.21.15.161:0/3298062321 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a340052d0 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a42ffd640 1 -- 172.21.15.161:0/3298062321 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a34005450 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a4c0c8ad0 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a42ffd640 1 -- 172.21.15.161:0/3298062321 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a34007260 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.920+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a4c0c9bd0 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.921+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a4c0d1450 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.921+0000 7f0a42ffd640 1 -- 172.21.15.161:0/3298062321 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0a340073c0 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.922+0000 7f0a42ffd640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a2c06b810 0x7f0a2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.922+0000 7f0a42ffd640 1 -- 172.21.15.161:0/3298062321 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0a34089540 con 0x7f0a4c0d2b40 2024-03-23T14:35:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.922+0000 7f0a4a59c640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a2c06b810 0x7f0a2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:37.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.922+0000 7f0a4a59c640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a2c06b810 0x7f0a2c06dcd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c036000 tx=0x7f0a3c03b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:37.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:37.930+0000 7f0a42ffd640 1 -- 172.21.15.161:0/3298062321 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0a4c0d1450 con 0x7f0a4c0d2b40 2024-03-23T14:35:38.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.104+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0a4c0d1450 con 0x7f0a4c0d2b40 2024-03-23T14:35:38.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.106+0000 7f0a42ffd640 1 -- 172.21.15.161:0/3298062321 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f0a4c0d1450 con 0x7f0a4c0d2b40 2024-03-23T14:35:38.102 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:38.102 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a2c06b810 msgr2=0x7f0a2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a2c06b810 0x7f0a2c06dcd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f0a3c036000 tx=0x7f0a3c03b040 comp rx=0 tx=0).stop 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c0d2b40 msgr2=0x7f0a4c0c8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c0d2b40 0x7f0a4c0c8ff0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f0a3400c400 tx=0x7f0a3400c8c0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 shutdown_connections 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0a2c06b810 0x7f0a2c06dcd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0a4c15e400 0x7f0a4c0b8630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0a4c0d2b40 0x7f0a4c0c8ff0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 --2- 172.21.15.161:0/3298062321 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0a4c0cf9d0 0x7f0a4c157100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 >> 172.21.15.161:0/3298062321 conn(0x7f0a4c09d190 msgr2=0x7f0a4c0b1230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 shutdown_connections 2024-03-23T14:35:38.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.108+0000 7f0a50e07640 1 -- 172.21.15.161:0/3298062321 wait complete. 2024-03-23T14:35:38.115 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:38.115 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:38.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.225+0000 7ff4e7581640 1 Processor -- start 2024-03-23T14:35:38.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.225+0000 7ff4e7581640 1 -- start start 2024-03-23T14:35:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e7581640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4e0161820 0x7ff4e00d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e7581640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d2ba0 0x7ff4e00b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e7581640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4e0161c00 con 0x7ff4e00d1270 2024-03-23T14:35:38.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e7581640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff4e00cea30 con 0x7ff4e0161820 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e7581640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e00cfd70 con 0x7ff4e00d2ba0 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e531d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4e0161820 0x7ff4e00d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e531d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4e0161820 0x7ff4e00d47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49780/0 (socket says 172.21.15.161:49780) 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e531d640 1 -- 172.21.15.161:0/3003887250 learned_addr learned my addr 172.21.15.161:0/3003887250 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e5b1e640 1 --2- 172.21.15.161:0/3003887250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d2ba0 0x7ff4e00b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4d7fff640 1 -- 172.21.15.161:0/3003887250 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1179420688 0 0) 0x7ff4e0161c00 con 0x7ff4e00d1270 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4d7fff640 1 -- 172.21.15.161:0/3003887250 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff4cc003520 con 0x7ff4e00d1270 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e531d640 1 -- 172.21.15.161:0/3003887250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d2ba0 msgr2=0x7ff4e00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e531d640 1 --2- 172.21.15.161:0/3003887250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d2ba0 0x7ff4e00b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e531d640 1 -- 172.21.15.161:0/3003887250 >> v1:172.21.15.183:6789/0 conn(0x7ff4e00d1270 legacy=0x7ff4e00d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.226+0000 7ff4e531d640 1 -- 172.21.15.161:0/3003887250 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e00cecd0 con 0x7ff4e0161820 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.227+0000 7ff4e531d640 1 --2- 172.21.15.161:0/3003887250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e00d47e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009870 tx=0x7ff4dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3f69a155624a291 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.227+0000 7ff4d7fff640 1 -- 172.21.15.161:0/3003887250 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc030840 con 0x7ff4e0161820 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.227+0000 7ff4d7fff640 1 -- 172.21.15.161:0/3003887250 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4dc0309a0 con 0x7ff4e0161820 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.227+0000 7ff4d7fff640 1 -- 172.21.15.161:0/3003887250 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4dc030d30 con 0x7ff4e0161820 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.228+0000 7ff4e7581640 1 -- 172.21.15.161:0/3003887250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 msgr2=0x7ff4e00d47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.228+0000 7ff4e7581640 1 --2- 172.21.15.161:0/3003887250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e00d47e0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc009870 tx=0x7ff4dc02b980 comp rx=0 tx=0).stop 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.228+0000 7ff4e7581640 1 -- 172.21.15.161:0/3003887250 shutdown_connections 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.228+0000 7ff4e7581640 1 --2- 172.21.15.161:0/3003887250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d2ba0 0x7ff4e00b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.228+0000 7ff4e7581640 1 --2- 172.21.15.161:0/3003887250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e00d47e0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.228+0000 7ff4e7581640 1 -- 172.21.15.161:0/3003887250 >> 172.21.15.161:0/3003887250 conn(0x7ff4e009d190 msgr2=0x7ff4e00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.228+0000 7ff4e7581640 1 -- 172.21.15.161:0/3003887250 shutdown_connections 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.228+0000 7ff4e7581640 1 -- 172.21.15.161:0/3003887250 wait complete. 2024-03-23T14:35:38.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.229+0000 7ff4e7581640 1 Processor -- start 2024-03-23T14:35:38.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.229+0000 7ff4e7581640 1 -- start start 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e7581640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d1270 0x7ff4e00a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e7581640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4e00d2ba0 0x7ff4e00cce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e7581640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e009ea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e7581640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01f8260 con 0x7ff4e0161820 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e7581640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff4e01f7300 con 0x7ff4e00d2ba0 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e7581640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff4e01f75e0 con 0x7ff4e00d1270 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e009ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e009ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49794/0 (socket says 172.21.15.161:49794) 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 -- 172.21.15.161:0/133452761 learned_addr learned my addr 172.21.15.161:0/133452761 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e531d640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d1270 0x7ff4e00a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e4b1c640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4e00d2ba0 0x7ff4e00cce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 -- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d1270 msgr2=0x7ff4e00a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d1270 0x7ff4e00a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 -- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4e00d2ba0 msgr2=0x7ff4e00cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4e00d2ba0 0x7ff4e00cce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 -- 172.21.15.161:0/133452761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4dc0094d0 con 0x7ff4e0161820 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4e5b1e640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e009ea80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800efc0 tx=0x7ff4d800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4d5ffb640 1 -- 172.21.15.161:0/133452761 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8014480 con 0x7ff4e0161820 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.230+0000 7ff4d5ffb640 1 -- 172.21.15.161:0/133452761 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d800f040 con 0x7ff4e0161820 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.231+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4e01f78c0 con 0x7ff4e0161820 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.231+0000 7ff4d5ffb640 1 -- 172.21.15.161:0/133452761 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d8014820 con 0x7ff4e0161820 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.231+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4e01f7e80 con 0x7ff4e0161820 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.232+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4e015db90 con 0x7ff4e0161820 2024-03-23T14:35:38.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.232+0000 7ff4d5ffb640 1 -- 172.21.15.161:0/133452761 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff4d8014980 con 0x7ff4e0161820 2024-03-23T14:35:38.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.232+0000 7ff4d5ffb640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c406b810 0x7ff4c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.233+0000 7ff4d5ffb640 1 -- 172.21.15.161:0/133452761 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7ff4d808ac40 con 0x7ff4e0161820 2024-03-23T14:35:38.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.233+0000 7ff4e531d640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c406b810 0x7ff4c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.233+0000 7ff4e531d640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c406b810 0x7ff4c406dcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc036000 tx=0x7ff4dc008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.237+0000 7ff4d5ffb640 1 -- 172.21.15.161:0/133452761 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff4e015db90 con 0x7ff4e0161820 2024-03-23T14:35:38.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.415+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4e015db90 con 0x7ff4e0161820 2024-03-23T14:35:38.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.415+0000 7ff4d5ffb640 1 -- 172.21.15.161:0/133452761 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7ff4e015db90 con 0x7ff4e0161820 2024-03-23T14:35:38.411 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:38.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c406b810 msgr2=0x7ff4c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c406b810 0x7ff4c406dcd0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc036000 tx=0x7ff4dc008030 comp rx=0 tx=0).stop 2024-03-23T14:35:38.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 msgr2=0x7ff4e009ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e009ea80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff4d800efc0 tx=0x7ff4d800c490 comp rx=0 tx=0).stop 2024-03-23T14:35:38.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 shutdown_connections 2024-03-23T14:35:38.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c406b810 0x7ff4c406dcd0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4e0161820 0x7ff4e009ea80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4e00d2ba0 0x7ff4e00cce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 --2- 172.21.15.161:0/133452761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4e00d1270 0x7ff4e00a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 >> 172.21.15.161:0/133452761 conn(0x7ff4e009d190 msgr2=0x7ff4e00d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:38.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 shutdown_connections 2024-03-23T14:35:38.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.417+0000 7ff4e7581640 1 -- 172.21.15.161:0/133452761 wait complete. 2024-03-23T14:35:38.424 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:38.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.532+0000 7efd0836c640 1 Processor -- start 2024-03-23T14:35:38.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.532+0000 7efd0836c640 1 -- start start 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd0836c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efd000b4010 0x7efd000cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd0836c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd0015bb90 0x7efd000d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd0836c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd0015c1f0 con 0x7efd000b43f0 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd0836c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efd0006ef20 con 0x7efd000b4010 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd0836c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efd0015f730 con 0x7efd0015bb90 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd06108640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efd000b4010 0x7efd000cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd06108640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efd000b4010 0x7efd000cfee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49800/0 (socket says 172.21.15.161:49800) 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd06108640 1 -- 172.21.15.161:0/673716764 learned_addr learned my addr 172.21.15.161:0/673716764 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd06909640 1 --2- 172.21.15.161:0/673716764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd0015bb90 0x7efd000d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd06108640 1 -- 172.21.15.161:0/673716764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd0015bb90 msgr2=0x7efd000d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:38.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd06108640 1 --2- 172.21.15.161:0/673716764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd0015bb90 0x7efd000d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd06108640 1 -- 172.21.15.161:0/673716764 >> v1:172.21.15.183:6789/0 conn(0x7efd000b43f0 legacy=0x7efd0015b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.533+0000 7efd06108640 1 -- 172.21.15.161:0/673716764 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0015e970 con 0x7efd000b4010 2024-03-23T14:35:38.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.534+0000 7efd06108640 1 --2- 172.21.15.161:0/673716764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b4010 0x7efd000cfee0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00bb40 tx=0x7efcfc02e0c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14fde49bfd604047 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.534+0000 7efd05106640 1 -- 172.21.15.161:0/673716764 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc02ee70 con 0x7efd000b4010 2024-03-23T14:35:38.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.534+0000 7efd05106640 1 -- 172.21.15.161:0/673716764 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcfc036440 con 0x7efd000b4010 2024-03-23T14:35:38.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.534+0000 7efd05106640 1 -- 172.21.15.161:0/673716764 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc0367d0 con 0x7efd000b4010 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.535+0000 7efd0836c640 1 -- 172.21.15.161:0/673716764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b4010 msgr2=0x7efd000cfee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.535+0000 7efd0836c640 1 --2- 172.21.15.161:0/673716764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b4010 0x7efd000cfee0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00bb40 tx=0x7efcfc02e0c0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.535+0000 7efd0836c640 1 -- 172.21.15.161:0/673716764 shutdown_connections 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.535+0000 7efd0836c640 1 --2- 172.21.15.161:0/673716764 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd0015bb90 0x7efd000d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.535+0000 7efd0836c640 1 --2- 172.21.15.161:0/673716764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b4010 0x7efd000cfee0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.535+0000 7efd0836c640 1 -- 172.21.15.161:0/673716764 >> 172.21.15.161:0/673716764 conn(0x7efd0006b4a0 msgr2=0x7efd0006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.535+0000 7efd0836c640 1 -- 172.21.15.161:0/673716764 shutdown_connections 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.535+0000 7efd0836c640 1 -- 172.21.15.161:0/673716764 wait complete. 2024-03-23T14:35:38.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.536+0000 7efd0836c640 1 Processor -- start 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.536+0000 7efd0836c640 1 -- start start 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.536+0000 7efd0836c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd000b4010 0x7efd000cce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.536+0000 7efd0836c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b43f0 0x7efd0014b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd0836c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd0015bb90 0x7efd00071410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd0836c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd00071f80 con 0x7efd000b43f0 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd0836c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efd000ae810 con 0x7efd0015bb90 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd0836c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efd0014fff0 con 0x7efd000b4010 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b43f0 0x7efd0014b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b43f0 0x7efd0014b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49816/0 (socket says 172.21.15.161:49816) 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 -- 172.21.15.161:0/1775698411 learned_addr learned my addr 172.21.15.161:0/1775698411 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd06108640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd000b4010 0x7efd000cce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd06909640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd0015bb90 0x7efd00071410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 -- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd000b4010 msgr2=0x7efd000cce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd000b4010 0x7efd000cce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.535 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 -- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd0015bb90 msgr2=0x7efd00071410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd0015bb90 0x7efd00071410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 -- 172.21.15.161:0/1775698411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcfc00b770 con 0x7efd000b43f0 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd06108640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd000b4010 0x7efd000cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd05907640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b43f0 0x7efd0014b830 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efcf000c3a0 tx=0x7efcf000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efcf6ffd640 1 -- 172.21.15.161:0/1775698411 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf00052d0 con 0x7efd000b43f0 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efcf6ffd640 1 -- 172.21.15.161:0/1775698411 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efcf0005450 con 0x7efd000b43f0 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd001502d0 con 0x7efd000b43f0 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efcf6ffd640 1 -- 172.21.15.161:0/1775698411 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcf0007260 con 0x7efd000b43f0 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.537+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd00150890 con 0x7efd000b43f0 2024-03-23T14:35:38.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.538+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd0006de90 con 0x7efd000b43f0 2024-03-23T14:35:38.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.539+0000 7efcf6ffd640 1 -- 172.21.15.161:0/1775698411 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efcf00073c0 con 0x7efd000b43f0 2024-03-23T14:35:38.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.539+0000 7efcf6ffd640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efce406b810 0x7efce406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.539+0000 7efcf6ffd640 1 -- 172.21.15.161:0/1775698411 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efcf0089530 con 0x7efd000b43f0 2024-03-23T14:35:38.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.539+0000 7efd06108640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efce406b810 0x7efce406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.540+0000 7efd06108640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efce406b810 0x7efce406dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efcfc038000 tx=0x7efcfc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.544+0000 7efcf6ffd640 1 -- 172.21.15.161:0/1775698411 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efcf0054000 con 0x7efd000b43f0 2024-03-23T14:35:38.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.714+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efd00073570 con 0x7efd000b43f0 2024-03-23T14:35:38.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.714+0000 7efcf6ffd640 1 -- 172.21.15.161:0/1775698411 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7efcf00576c0 con 0x7efd000b43f0 2024-03-23T14:35:38.710 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:38.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efce406b810 msgr2=0x7efce406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efce406b810 0x7efce406dcd0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7efcfc038000 tx=0x7efcfc008040 comp rx=0 tx=0).stop 2024-03-23T14:35:38.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b43f0 msgr2=0x7efd0014b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b43f0 0x7efd0014b830 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7efcf000c3a0 tx=0x7efcf000c860 comp rx=0 tx=0).stop 2024-03-23T14:35:38.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 shutdown_connections 2024-03-23T14:35:38.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efce406b810 0x7efce406dcd0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd0015bb90 0x7efd00071410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd000b43f0 0x7efd0014b830 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 --2- 172.21.15.161:0/1775698411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd000b4010 0x7efd000cce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 >> 172.21.15.161:0/1775698411 conn(0x7efd0006b4a0 msgr2=0x7efd00154100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:38.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 shutdown_connections 2024-03-23T14:35:38.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.716+0000 7efd0836c640 1 -- 172.21.15.161:0/1775698411 wait complete. 2024-03-23T14:35:38.723 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:38.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.832+0000 7fafd6bb5640 1 Processor -- start 2024-03-23T14:35:38.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.833+0000 7fafd6bb5640 1 -- start start 2024-03-23T14:35:38.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.833+0000 7fafd6bb5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd015f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.833+0000 7fafd6bb5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafd00cd3e0 con 0x7fafd00724b0 2024-03-23T14:35:38.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.833+0000 7fafd6bb5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafd00cce20 con 0x7fafd006b2f0 2024-03-23T14:35:38.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.833+0000 7fafd6bb5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fafd00cd100 con 0x7fafd006b6d0 2024-03-23T14:35:38.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.833+0000 7fafd5152640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fafd00724b0 0x7fafd015f980 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47992/0 (socket says 172.21.15.161:47992) 2024-03-23T14:35:38.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.833+0000 7fafd5152640 1 -- 172.21.15.161:0/3004506 learned_addr learned my addr 172.21.15.161:0/3004506 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:38.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.833+0000 7fafcffff640 1 --2- 172.21.15.161:0/3004506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd015f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.834+0000 7fafcf7fe640 1 -- 172.21.15.161:0/3004506 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3218295874 0 0) 0x7fafd00cd3e0 con 0x7fafd00724b0 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.834+0000 7fafcf7fe640 1 -- 172.21.15.161:0/3004506 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafbc003520 con 0x7fafd00724b0 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.834+0000 7fafcf7fe640 1 -- 172.21.15.161:0/3004506 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2669680011 0 0) 0x7fafd00cce20 con 0x7fafd006b2f0 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.834+0000 7fafcf7fe640 1 -- 172.21.15.161:0/3004506 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafd00cd3e0 con 0x7fafd006b2f0 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.834+0000 7fafcffff640 1 -- 172.21.15.161:0/3004506 >> v1:172.21.15.183:6789/0 conn(0x7fafd006b2f0 legacy=0x7fafd00cc5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.834+0000 7fafcffff640 1 -- 172.21.15.161:0/3004506 >> v1:172.21.15.161:6789/0 conn(0x7fafd00724b0 legacy=0x7fafd015f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.834+0000 7fafcffff640 1 -- 172.21.15.161:0/3004506 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafd0071cb0 con 0x7fafd006b6d0 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafcffff640 1 --2- 172.21.15.161:0/3004506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd015f0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097d0 tx=0x7fafb802b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8bfa384401216ac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafcf7fe640 1 -- 172.21.15.161:0/3004506 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8031800 con 0x7fafd006b6d0 2024-03-23T14:35:38.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafcf7fe640 1 -- 172.21.15.161:0/3004506 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafb8031960 con 0x7fafd006b6d0 2024-03-23T14:35:38.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafcf7fe640 1 -- 172.21.15.161:0/3004506 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafb8031cd0 con 0x7fafd006b6d0 2024-03-23T14:35:38.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafd6bb5640 1 -- 172.21.15.161:0/3004506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 msgr2=0x7fafd015f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafd6bb5640 1 --2- 172.21.15.161:0/3004506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd015f0c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fafb80097d0 tx=0x7fafb802b8a0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafd6bb5640 1 -- 172.21.15.161:0/3004506 shutdown_connections 2024-03-23T14:35:38.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafd6bb5640 1 --2- 172.21.15.161:0/3004506 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd015f0c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafd6bb5640 1 -- 172.21.15.161:0/3004506 >> 172.21.15.161:0/3004506 conn(0x7fafd00d0b10 msgr2=0x7fafd00d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:38.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafd6bb5640 1 -- 172.21.15.161:0/3004506 shutdown_connections 2024-03-23T14:35:38.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.835+0000 7fafd6bb5640 1 -- 172.21.15.161:0/3004506 wait complete. 2024-03-23T14:35:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.836+0000 7fafd6bb5640 1 Processor -- start 2024-03-23T14:35:38.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.836+0000 7fafd6bb5640 1 -- start start 2024-03-23T14:35:38.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.837+0000 7fafd6bb5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd006b2f0 0x7fafd01f73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.837+0000 7fafd6bb5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd00a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.837+0000 7fafd6bb5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd00724b0 0x7fafd015d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.837+0000 7fafd6bb5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fafd01f7930 con 0x7fafd00724b0 2024-03-23T14:35:38.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.837+0000 7fafd6bb5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fafd00a1150 con 0x7fafd006b2f0 2024-03-23T14:35:38.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.837+0000 7fafd6bb5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fafd00a1430 con 0x7fafd006b6d0 2024-03-23T14:35:38.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.837+0000 7fafd5152640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd00724b0 0x7fafd015d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.837+0000 7fafd5152640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd00724b0 0x7fafd015d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49828/0 (socket says 172.21.15.161:49828) 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd5152640 1 -- 172.21.15.161:0/4227453822 learned_addr learned my addr 172.21.15.161:0/4227453822 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafcffff640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd00a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd4951640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd006b2f0 0x7fafd01f73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd5152640 1 -- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 msgr2=0x7fafd00a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd5152640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd00a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd5152640 1 -- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd006b2f0 msgr2=0x7fafd01f73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd5152640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd006b2f0 0x7fafd01f73f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd5152640 1 -- 172.21.15.161:0/4227453822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafb8009400 con 0x7fafd00724b0 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd5152640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd00724b0 0x7fafd015d4a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c390 tx=0x7fafc800c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafcd7fa640 1 -- 172.21.15.161:0/4227453822 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8005ce0 con 0x7fafd00724b0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafcd7fa640 1 -- 172.21.15.161:0/4227453822 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafc8005e60 con 0x7fafd00724b0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafd00a1710 con 0x7fafd00724b0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafcd7fa640 1 -- 172.21.15.161:0/4227453822 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc8007310 con 0x7fafd00724b0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.838+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafd00c4640 con 0x7fafd00724b0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.839+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafd00a1950 con 0x7fafd00724b0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.839+0000 7fafcd7fa640 1 -- 172.21.15.161:0/4227453822 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fafc8007470 con 0x7fafd00724b0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.840+0000 7fafcd7fa640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fafb006b810 0x7fafb006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.840+0000 7fafcd7fa640 1 -- 172.21.15.161:0/4227453822 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fafc8089760 con 0x7fafd00724b0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.840+0000 7fafd4951640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fafb006b810 0x7fafb006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:38.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.840+0000 7fafd4951640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fafb006b810 0x7fafb006dcd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fafc0001ea0 tx=0x7fafc0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:38.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:38.844+0000 7fafcd7fa640 1 -- 172.21.15.161:0/4227453822 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fafd00a1950 con 0x7fafd00724b0 2024-03-23T14:35:39.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.014+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fafd009ee50 con 0x7fafd00724b0 2024-03-23T14:35:39.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.014+0000 7fafcd7fa640 1 -- 172.21.15.161:0/4227453822 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7fafd009ee50 con 0x7fafd00724b0 2024-03-23T14:35:39.011 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:39.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fafb006b810 msgr2=0x7fafb006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fafb006b810 0x7fafb006dcd0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fafc0001ea0 tx=0x7fafc0009040 comp rx=0 tx=0).stop 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd00724b0 msgr2=0x7fafd015d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd00724b0 0x7fafd015d4a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fafc800c390 tx=0x7fafc800c850 comp rx=0 tx=0).stop 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 shutdown_connections 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fafb006b810 0x7fafb006dcd0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fafd00724b0 0x7fafd015d4a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fafd006b6d0 0x7fafd00a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 --2- 172.21.15.161:0/4227453822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fafd006b2f0 0x7fafd01f73f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 >> 172.21.15.161:0/4227453822 conn(0x7fafd00d0b10 msgr2=0x7fafd0160390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 shutdown_connections 2024-03-23T14:35:39.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.016+0000 7fafd6bb5640 1 -- 172.21.15.161:0/4227453822 wait complete. 2024-03-23T14:35:39.023 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-23T14:35:39.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.128+0000 7f311a743640 1 Processor -- start 2024-03-23T14:35:39.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.128+0000 7f311a743640 1 -- start start 2024-03-23T14:35:39.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f311a743640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f311406eb30 0x7f31140b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f311a743640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b3610 0x7f31140b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f311a743640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3114070b90 con 0x7f31140b4260 2024-03-23T14:35:39.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f311a743640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f31140b3fe0 con 0x7f311406eb30 2024-03-23T14:35:39.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f311a743640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f311415ab90 con 0x7f31140b3610 2024-03-23T14:35:39.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f3113fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f311406eb30 0x7f31140b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f3118ce0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b3610 0x7f31140b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f31137fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f31140b4260 0x7f311415b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:32794/0 (socket says 172.21.15.161:32794) 2024-03-23T14:35:39.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f31137fe640 1 -- 172.21.15.161:0/730368168 learned_addr learned my addr 172.21.15.161:0/730368168 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:39.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f3113fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f311406eb30 0x7f31140b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:38820/0 (socket says 172.21.15.161:38820) 2024-03-23T14:35:39.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f3113fff640 1 -- 172.21.15.161:0/730368168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b3610 msgr2=0x7f31140b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:39.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f3113fff640 1 --2- 172.21.15.161:0/730368168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b3610 0x7f31140b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f3113fff640 1 -- 172.21.15.161:0/730368168 >> v1:172.21.15.183:6789/0 conn(0x7f31140b4260 legacy=0x7f311415b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f3113fff640 1 -- 172.21.15.161:0/730368168 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f311415b0d0 con 0x7f311406eb30 2024-03-23T14:35:39.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.129+0000 7f3112ffd640 1 -- 172.21.15.161:0/730368168 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 96256411 0 0) 0x7f3114070b90 con 0x7f31140b4260 2024-03-23T14:35:39.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.130+0000 7f3113fff640 1 --2- 172.21.15.161:0/730368168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311406eb30 0x7f31140b4870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f31040097a0 tx=0x7f310402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a9b2b85401feff8a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:39.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.130+0000 7f3112ffd640 1 -- 172.21.15.161:0/730368168 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f310402e070 con 0x7f311406eb30 2024-03-23T14:35:39.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.130+0000 7f3112ffd640 1 -- 172.21.15.161:0/730368168 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31040315a0 con 0x7f311406eb30 2024-03-23T14:35:39.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.130+0000 7f3112ffd640 1 -- 172.21.15.161:0/730368168 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104031930 con 0x7f311406eb30 2024-03-23T14:35:39.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.130+0000 7f311a743640 1 -- 172.21.15.161:0/730368168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311406eb30 msgr2=0x7f31140b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.130+0000 7f311a743640 1 --2- 172.21.15.161:0/730368168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311406eb30 0x7f31140b4870 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f31040097a0 tx=0x7f310402b840 comp rx=0 tx=0).stop 2024-03-23T14:35:39.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.131+0000 7f311a743640 1 -- 172.21.15.161:0/730368168 shutdown_connections 2024-03-23T14:35:39.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.131+0000 7f311a743640 1 --2- 172.21.15.161:0/730368168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b3610 0x7f31140b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.131+0000 7f311a743640 1 --2- 172.21.15.161:0/730368168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f311406eb30 0x7f31140b4870 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.131+0000 7f311a743640 1 -- 172.21.15.161:0/730368168 >> 172.21.15.161:0/730368168 conn(0x7f31140b17f0 msgr2=0x7f31140b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:39.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.131+0000 7f311a743640 1 -- 172.21.15.161:0/730368168 shutdown_connections 2024-03-23T14:35:39.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.131+0000 7f311a743640 1 -- 172.21.15.161:0/730368168 wait complete. 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.132+0000 7f311a743640 1 Processor -- start 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.132+0000 7f311a743640 1 -- start start 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f311a743640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f311406eb30 0x7f31140cde20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f311a743640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31140b3610 0x7f311414d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f311a743640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b4260 0x7f3114146690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f311a743640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f311414b990 con 0x7f311406eb30 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f311a743640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f31140cd990 con 0x7f31140b3610 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f311a743640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f31140ccbc0 con 0x7f31140b4260 2024-03-23T14:35:39.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f311406eb30 0x7f31140cde20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f311406eb30 0x7f31140cde20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49836/0 (socket says 172.21.15.161:49836) 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 -- 172.21.15.161:0/2415080612 learned_addr learned my addr 172.21.15.161:0/2415080612 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f31137fe640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31140b3610 0x7f311414d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3118ce0640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b4260 0x7f3114146690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 -- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b4260 msgr2=0x7f3114146690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b4260 0x7f3114146690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 -- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31140b3610 msgr2=0x7f311414d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31140b3610 0x7f311414d1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 -- 172.21.15.161:0/2415080612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3104009400 con 0x7f311406eb30 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3113fff640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f311406eb30 0x7f31140cde20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3104034040 tx=0x7f31040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3110ff9640 1 -- 172.21.15.161:0/2415080612 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104032840 con 0x7f311406eb30 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3110ff9640 1 -- 172.21.15.161:0/2415080612 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31040329c0 con 0x7f311406eb30 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31140cd340 con 0x7f311406eb30 2024-03-23T14:35:39.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f3110ff9640 1 -- 172.21.15.161:0/2415080612 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3104032d70 con 0x7f311406eb30 2024-03-23T14:35:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.133+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31140ccea0 con 0x7f311406eb30 2024-03-23T14:35:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.134+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3114068370 con 0x7f311406eb30 2024-03-23T14:35:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.135+0000 7f3110ff9640 1 -- 172.21.15.161:0/2415080612 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f310404e430 con 0x7f311406eb30 2024-03-23T14:35:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.135+0000 7f3110ff9640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f310006b810 0x7f310006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.135+0000 7f3110ff9640 1 -- 172.21.15.161:0/2415080612 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f31040abfd0 con 0x7f311406eb30 2024-03-23T14:35:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.135+0000 7f31137fe640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f310006b810 0x7f310006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.136+0000 7f31137fe640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f310006b810 0x7f310006dcd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f311415d250 tx=0x7f3108007de0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:39.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.140+0000 7f3110ff9640 1 -- 172.21.15.161:0/2415080612 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3104076b30 con 0x7f311406eb30 2024-03-23T14:35:39.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.377+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7f311404f550 con 0x7f311406eb30 2024-03-23T14:35:39.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.821+0000 7f3110ff9640 1 -- 172.21.15.161:0/2415080612 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v46) ==== 150+0+0 (secure 0 0 0) 0x7f310407a1f0 con 0x7f311406eb30 2024-03-23T14:35:39.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f310006b810 msgr2=0x7f310006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f310006b810 0x7f310006dcd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f311415d250 tx=0x7f3108007de0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f311406eb30 msgr2=0x7f31140cde20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f311406eb30 0x7f31140cde20 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f3104034040 tx=0x7f31040046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 shutdown_connections 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f310006b810 0x7f310006dcd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f31140b4260 0x7f3114146690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f31140b3610 0x7f311414d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 --2- 172.21.15.161:0/2415080612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f311406eb30 0x7f31140cde20 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 >> 172.21.15.161:0/2415080612 conn(0x7f31140b17f0 msgr2=0x7f311415bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 shutdown_connections 2024-03-23T14:35:39.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.824+0000 7f311a743640 1 -- 172.21.15.161:0/2415080612 wait complete. 2024-03-23T14:35:39.832 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T14:35:39.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0587dc3640 1 Processor -- start 2024-03-23T14:35:39.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0587dc3640 1 -- start start 2024-03-23T14:35:39.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0587dc3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f05800d2cd0 0x7f05800ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0587dc3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05800d1cb0 0x7f05800cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0587dc3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05800d1330 con 0x7f0580161750 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0587dc3640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f05800d0720 con 0x7f05800d1cb0 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0587dc3640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f05800d0a00 con 0x7f05800d2cd0 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0586360640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05800d1cb0 0x7f05800cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0585b5f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0580161750 0x7f05800d25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:47998/0 (socket says 172.21.15.161:47998) 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0586360640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05800d1cb0 0x7f05800cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49848/0 (socket says 172.21.15.161:49848) 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.940+0000 7f0585b5f640 1 -- 172.21.15.161:0/481855690 learned_addr learned my addr 172.21.15.161:0/481855690 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.941+0000 7f058535e640 1 --2- 172.21.15.161:0/481855690 >> v2:172.21.15.183:3300/0 conn(0x7f05800d2cd0 0x7f05800ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.941+0000 7f0586360640 1 -- 172.21.15.161:0/481855690 >> v2:172.21.15.183:3300/0 conn(0x7f05800d2cd0 msgr2=0x7f05800ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.941+0000 7f0586360640 1 --2- 172.21.15.161:0/481855690 >> v2:172.21.15.183:3300/0 conn(0x7f05800d2cd0 0x7f05800ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.941+0000 7f0586360640 1 -- 172.21.15.161:0/481855690 >> v1:172.21.15.161:6789/0 conn(0x7f0580161750 legacy=0x7f05800d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.941+0000 7f0586360640 1 -- 172.21.15.161:0/481855690 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05800d0ce0 con 0x7f05800d1cb0 2024-03-23T14:35:39.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.941+0000 7f0586360640 1 --2- 172.21.15.161:0/481855690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05800cf050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f05800c7be0 tx=0x7f057802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4385b7ff71cff891 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:39.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0584b5d640 1 -- 172.21.15.161:0/481855690 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578031830 con 0x7f05800d1cb0 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0584b5d640 1 -- 172.21.15.161:0/481855690 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0578031990 con 0x7f05800d1cb0 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0584b5d640 1 -- 172.21.15.161:0/481855690 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0578031d20 con 0x7f05800d1cb0 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0587dc3640 1 -- 172.21.15.161:0/481855690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 msgr2=0x7f05800cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0587dc3640 1 --2- 172.21.15.161:0/481855690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05800cf050 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f05800c7be0 tx=0x7f057802b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0587dc3640 1 -- 172.21.15.161:0/481855690 shutdown_connections 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0587dc3640 1 --2- 172.21.15.161:0/481855690 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05800cf050 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0587dc3640 1 --2- 172.21.15.161:0/481855690 >> v2:172.21.15.183:3300/0 conn(0x7f05800d2cd0 0x7f05800ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0587dc3640 1 -- 172.21.15.161:0/481855690 >> 172.21.15.161:0/481855690 conn(0x7f058009d450 msgr2=0x7f05800d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0587dc3640 1 -- 172.21.15.161:0/481855690 shutdown_connections 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.942+0000 7f0587dc3640 1 -- 172.21.15.161:0/481855690 wait complete. 2024-03-23T14:35:39.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.943+0000 7f0587dc3640 1 Processor -- start 2024-03-23T14:35:39.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.943+0000 7f0587dc3640 1 -- start start 2024-03-23T14:35:39.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0587dc3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05801f4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0587dc3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05800d2cd0 0x7f058015cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0587dc3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0580161750 0x7f05800a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0587dc3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f05800cbe90 con 0x7f05800d1cb0 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0587dc3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f058015d4d0 con 0x7f05800d2cd0 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0587dc3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f05800cb250 con 0x7f0580161750 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0585b5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05801f4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0585b5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05801f4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49858/0 (socket says 172.21.15.161:49858) 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0585b5f640 1 -- 172.21.15.161:0/677500040 learned_addr learned my addr 172.21.15.161:0/677500040 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f058535e640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05800d2cd0 0x7f058015cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.944+0000 7f0586360640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0580161750 0x7f05800a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f0585b5f640 1 -- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0580161750 msgr2=0x7f05800a16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f0585b5f640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0580161750 0x7f05800a16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f0585b5f640 1 -- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05800d2cd0 msgr2=0x7f058015cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:39.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f0585b5f640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05800d2cd0 0x7f058015cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f0585b5f640 1 -- 172.21.15.161:0/677500040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0578009400 con 0x7f05800d1cb0 2024-03-23T14:35:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f0585b5f640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05801f4a70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f057c00ca70 tx=0x7f057c00cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f05767fc640 1 -- 172.21.15.161:0/677500040 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c00abd0 con 0x7f05800d1cb0 2024-03-23T14:35:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f05767fc640 1 -- 172.21.15.161:0/677500040 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f057c00ad30 con 0x7f05800d1cb0 2024-03-23T14:35:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f05767fc640 1 -- 172.21.15.161:0/677500040 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f057c012620 con 0x7f05800d1cb0 2024-03-23T14:35:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05800cb4f0 con 0x7f05800d1cb0 2024-03-23T14:35:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.945+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05800cbab0 con 0x7f05800d1cb0 2024-03-23T14:35:39.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.946+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05800a2030 con 0x7f05800d1cb0 2024-03-23T14:35:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.946+0000 7f05767fc640 1 -- 172.21.15.161:0/677500040 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f057c007070 con 0x7f05800d1cb0 2024-03-23T14:35:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.947+0000 7f05767fc640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f056406b810 0x7f056406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.947+0000 7f05767fc640 1 -- 172.21.15.161:0/677500040 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f057c08a030 con 0x7f05800d1cb0 2024-03-23T14:35:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.947+0000 7f058535e640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f056406b810 0x7f056406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:39.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.947+0000 7f058535e640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f056406b810 0x7f056406dcd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f05800b8d40 tx=0x7f0570009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:39.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:39.951+0000 7f05767fc640 1 -- 172.21.15.161:0/677500040 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f057c054b90 con 0x7f05800d1cb0 2024-03-23T14:35:40.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.121+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f058004f550 con 0x7f05800d1cb0 2024-03-23T14:35:40.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.127+0000 7f05767fc640 1 -- 172.21.15.161:0/677500040 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) ==== 122+0+0 (secure 0 0 0) 0x7f057c0170c0 con 0x7f05800d1cb0 2024-03-23T14:35:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f056406b810 msgr2=0x7f056406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f056406b810 0x7f056406dcd0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f05800b8d40 tx=0x7f0570009190 comp rx=0 tx=0).stop 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 msgr2=0x7f05801f4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05801f4a70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f057c00ca70 tx=0x7f057c00cf30 comp rx=0 tx=0).stop 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 shutdown_connections 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f056406b810 0x7f056406dcd0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0580161750 0x7f05800a16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05800d2cd0 0x7f058015cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 --2- 172.21.15.161:0/677500040 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05800d1cb0 0x7f05801f4a70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 >> 172.21.15.161:0/677500040 conn(0x7f058009d450 msgr2=0x7f05800b2660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 shutdown_connections 2024-03-23T14:35:40.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.129+0000 7f0587dc3640 1 -- 172.21.15.161:0/677500040 wait complete. 2024-03-23T14:35:40.136 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T14:35:40.136 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T14:35:40.220 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T14:35:40.257 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T14:35:40.257 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:35:40.257 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:40.278 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:35:40.278 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:40.334 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:40.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.480+0000 7f8531ff4640 1 Processor -- start 2024-03-23T14:35:40.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.480+0000 7f8531ff4640 1 -- start start 2024-03-23T14:35:40.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.480+0000 7f8531ff4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f852c15e400 0x7f852c0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f8531ff4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f852c0d0460 con 0x7f852c0d2b40 2024-03-23T14:35:40.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f8531ff4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f852c0d06b0 con 0x7f852c0cf9d0 2024-03-23T14:35:40.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f8531ff4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f852c0d1050 con 0x7f852c15e400 2024-03-23T14:35:40.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f852b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f852c15e400 0x7f852c0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f852bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f852c0d2b40 0x7f852c0b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48010/0 (socket says 172.21.15.161:48010) 2024-03-23T14:35:40.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f852bfff640 1 -- 172.21.15.161:0/3485952239 learned_addr learned my addr 172.21.15.161:0/3485952239 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:40.477 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f852b7fe640 1 -- 172.21.15.161:0/3485952239 >> v1:172.21.15.183:6789/0 conn(0x7f852c0cf9d0 legacy=0x7f852c0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f852b7fe640 1 -- 172.21.15.161:0/3485952239 >> v1:172.21.15.161:6789/0 conn(0x7f852c0d2b40 legacy=0x7f852c0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f852b7fe640 1 -- 172.21.15.161:0/3485952239 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852c0d0990 con 0x7f852c15e400 2024-03-23T14:35:40.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.481+0000 7f852a7fc640 1 -- 172.21.15.161:0/3485952239 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4053528022 0 0) 0x7f852c0d0460 con 0x7f852c0d2b40 2024-03-23T14:35:40.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f852b7fe640 1 --2- 172.21.15.161:0/3485952239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0cf490 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f851c0097a0 tx=0x7f851c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=715a915d9c929a0b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:40.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f852a7fc640 1 -- 172.21.15.161:0/3485952239 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c030610 con 0x7f852c15e400 2024-03-23T14:35:40.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f852a7fc640 1 -- 172.21.15.161:0/3485952239 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f851c030770 con 0x7f852c15e400 2024-03-23T14:35:40.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f852a7fc640 1 -- 172.21.15.161:0/3485952239 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f851c030b00 con 0x7f852c15e400 2024-03-23T14:35:40.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f8531ff4640 1 -- 172.21.15.161:0/3485952239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 msgr2=0x7f852c0cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f8531ff4640 1 --2- 172.21.15.161:0/3485952239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0cf490 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f851c0097a0 tx=0x7f851c02b960 comp rx=0 tx=0).stop 2024-03-23T14:35:40.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f8531ff4640 1 -- 172.21.15.161:0/3485952239 shutdown_connections 2024-03-23T14:35:40.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f8531ff4640 1 --2- 172.21.15.161:0/3485952239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0cf490 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f8531ff4640 1 -- 172.21.15.161:0/3485952239 >> 172.21.15.161:0/3485952239 conn(0x7f852c09d190 msgr2=0x7f852c0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:40.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f8531ff4640 1 -- 172.21.15.161:0/3485952239 shutdown_connections 2024-03-23T14:35:40.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.482+0000 7f8531ff4640 1 -- 172.21.15.161:0/3485952239 wait complete. 2024-03-23T14:35:40.480 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.484+0000 7f8531ff4640 1 Processor -- start 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.484+0000 7f8531ff4640 1 -- start start 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8531ff4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c0cf9d0 0x7f852c0cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8531ff4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f852c0d2b40 0x7f852c153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8531ff4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8531ff4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f852c0cbea0 con 0x7f852c15e400 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8531ff4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f852c0ccb70 con 0x7f852c0cf9d0 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8531ff4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f852c153b90 con 0x7f852c0d2b40 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0b0e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:49888/0 (socket says 172.21.15.161:49888) 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 -- 172.21.15.161:0/1371873991 learned_addr learned my addr 172.21.15.161:0/1371873991 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:40.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852affd640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f852c0d2b40 0x7f852c153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852b7fe640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c0cf9d0 0x7f852c0cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 -- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f852c0d2b40 msgr2=0x7f852c153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f852c0d2b40 0x7f852c153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 -- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c0cf9d0 msgr2=0x7f852c0cc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c0cf9d0 0x7f852c0cc630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 -- 172.21.15.161:0/1371873991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851c009400 con 0x7f852c15e400 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852b7fe640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c0cf9d0 0x7f852c0cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f852bfff640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0b0e70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f852400c7b0 tx=0x7f852400cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8513fff640 1 -- 172.21.15.161:0/1371873991 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852400a640 con 0x7f852c15e400 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8513fff640 1 -- 172.21.15.161:0/1371873991 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f852400a7a0 con 0x7f852c15e400 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852c0cb290 con 0x7f852c15e400 2024-03-23T14:35:40.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8513fff640 1 -- 172.21.15.161:0/1371873991 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f852400ab90 con 0x7f852c15e400 2024-03-23T14:35:40.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.485+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f852c0cb850 con 0x7f852c15e400 2024-03-23T14:35:40.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.486+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f852c0d1450 con 0x7f852c15e400 2024-03-23T14:35:40.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.487+0000 7f8513fff640 1 -- 172.21.15.161:0/1371873991 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f852400acf0 con 0x7f852c15e400 2024-03-23T14:35:40.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.487+0000 7f8513fff640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.487+0000 7f8513fff640 1 -- 172.21.15.161:0/1371873991 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f85240898d0 con 0x7f852c15e400 2024-03-23T14:35:40.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.487+0000 7f852b7fe640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.488+0000 7f852b7fe640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f851c036000 tx=0x7f851c008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:40.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.491+0000 7f8513fff640 1 -- 172.21.15.161:0/1371873991 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f852c0d1450 con 0x7f852c15e400 2024-03-23T14:35:40.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.663+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f852c0d1450 con 0x7f852c15e400 2024-03-23T14:35:40.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.664+0000 7f8513fff640 1 -- 172.21.15.161:0/1371873991 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f852c0d1450 con 0x7f852c15e400 2024-03-23T14:35:40.661 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:40.661 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.666+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 msgr2=0x7f851806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.666+0000 7f8531ff4640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f851c036000 tx=0x7f851c008030 comp rx=0 tx=0).stop 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 msgr2=0x7f852c0b0e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0b0e70 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f852400c7b0 tx=0x7f852400cc70 comp rx=0 tx=0).stop 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 shutdown_connections 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c15e400 0x7f852c0b0e70 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f852c0d2b40 0x7f852c153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 --2- 172.21.15.161:0/1371873991 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c0cf9d0 0x7f852c0cc630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 >> 172.21.15.161:0/1371873991 conn(0x7f852c09d190 msgr2=0x7f852c161a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:40.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 shutdown_connections 2024-03-23T14:35:40.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.667+0000 7f8531ff4640 1 -- 172.21.15.161:0/1371873991 wait complete. 2024-03-23T14:35:40.674 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:40.674 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:40.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.783+0000 7f7802377640 1 Processor -- start 2024-03-23T14:35:40.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.783+0000 7f7802377640 1 -- start start 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f7802377640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f77fc0d0380 0x7f77fc165b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f7802377640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77fc15f480 con 0x7f77fc1552f0 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f7802377640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77fc1556d0 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f7802377640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f77fc06eee0 con 0x7f77fc0d0380 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f7800914640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f77fc0d0380 0x7f77fc165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f77fbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f77fc1552f0 0x7f77fc0b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60926/0 (socket says 172.21.15.161:60926) 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f77fbfff640 1 -- 172.21.15.161:0/4023503601 learned_addr learned my addr 172.21.15.161:0/4023503601 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f7800914640 1 -- 172.21.15.161:0/4023503601 >> v1:172.21.15.183:6789/0 conn(0x7f77fc0b2aa0 legacy=0x7f77fc0cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f7800914640 1 -- 172.21.15.161:0/4023503601 >> v1:172.21.15.161:6789/0 conn(0x7f77fc1552f0 legacy=0x7f77fc0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f77faffd640 1 -- 172.21.15.161:0/4023503601 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4117056959 0 0) 0x7f77fc15f480 con 0x7f77fc1552f0 2024-03-23T14:35:40.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.784+0000 7f7800914640 1 -- 172.21.15.161:0/4023503601 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77fc15e870 con 0x7f77fc0d0380 2024-03-23T14:35:40.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f7800914640 1 --2- 172.21.15.161:0/4023503601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0d0380 0x7f77fc165b60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f77f4009d90 tx=0x7f77f402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7a3cce5c73563fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:40.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f77faffd640 1 -- 172.21.15.161:0/4023503601 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f402cd00 con 0x7f77fc0d0380 2024-03-23T14:35:40.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f77faffd640 1 -- 172.21.15.161:0/4023503601 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77f402ce60 con 0x7f77fc0d0380 2024-03-23T14:35:40.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f77faffd640 1 -- 172.21.15.161:0/4023503601 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77f4034660 con 0x7f77fc0d0380 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f7802377640 1 -- 172.21.15.161:0/4023503601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0d0380 msgr2=0x7f77fc165b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f7802377640 1 --2- 172.21.15.161:0/4023503601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0d0380 0x7f77fc165b60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f77f4009d90 tx=0x7f77f402bf50 comp rx=0 tx=0).stop 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f7802377640 1 -- 172.21.15.161:0/4023503601 shutdown_connections 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f7802377640 1 --2- 172.21.15.161:0/4023503601 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0d0380 0x7f77fc165b60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f7802377640 1 -- 172.21.15.161:0/4023503601 >> 172.21.15.161:0/4023503601 conn(0x7f77fc06b260 msgr2=0x7f77fc06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.785+0000 7f7802377640 1 -- 172.21.15.161:0/4023503601 shutdown_connections 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.786+0000 7f7802377640 1 -- 172.21.15.161:0/4023503601 wait complete. 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.787+0000 7f7802377640 1 Processor -- start 2024-03-23T14:35:40.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.787+0000 7f7802377640 1 -- start start 2024-03-23T14:35:40.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.787+0000 7f7802377640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0b2aa0 0x7f77fc1561e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.787+0000 7f7802377640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77fc0d0380 0x7f77fc1f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f7802377640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77fc1552f0 0x7f77fc1f4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f7802377640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc156a30 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f7802377640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f77fc158690 con 0x7f77fc0d0380 2024-03-23T14:35:40.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f7802377640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f77fc157e50 con 0x7f77fc1552f0 2024-03-23T14:35:40.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0b2aa0 0x7f77fc1561e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0b2aa0 0x7f77fc1561e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41372/0 (socket says 172.21.15.161:41372) 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 -- 172.21.15.161:0/1115973060 learned_addr learned my addr 172.21.15.161:0/1115973060 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f7800914640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77fc1552f0 0x7f77fc1f4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fb7fe640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77fc0d0380 0x7f77fc1f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 -- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77fc1552f0 msgr2=0x7f77fc1f4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77fc1552f0 0x7f77fc1f4bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 -- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77fc0d0380 msgr2=0x7f77fc1f3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77fc0d0380 0x7f77fc1f3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 -- 172.21.15.161:0/1115973060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77f4009a70 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77fbfff640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0b2aa0 0x7f77fc1561e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00c4a0 tx=0x7f77ec00c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77f8ff9640 1 -- 172.21.15.161:0/1115973060 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec0074e0 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77f8ff9640 1 -- 172.21.15.161:0/1115973060 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77ec007640 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77fc1580d0 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f77f8ff9640 1 -- 172.21.15.161:0/1115973060 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77ec004a00 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.788+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77fc15d0f0 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.789+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77fc1567e0 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.790+0000 7f77f8ff9640 1 -- 172.21.15.161:0/1115973060 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f77ec004b60 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.790+0000 7f77f8ff9640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f77e806b810 0x7f77e806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:40.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.790+0000 7f77f8ff9640 1 -- 172.21.15.161:0/1115973060 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f77ec0895f0 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.790+0000 7f77fb7fe640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f77e806b810 0x7f77e806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:40.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.791+0000 7f77fb7fe640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f77e806b810 0x7f77e806dcd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0cc260 tx=0x7f77f0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:40.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.794+0000 7f77f8ff9640 1 -- 172.21.15.161:0/1115973060 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f77ec054150 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.966+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f77fc04f4e0 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.966+0000 7f77f8ff9640 1 -- 172.21.15.161:0/1115973060 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f77ec057810 con 0x7f77fc0b2aa0 2024-03-23T14:35:40.963 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:40.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f77e806b810 msgr2=0x7f77e806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f77e806b810 0x7f77e806dcd0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f77fc0cc260 tx=0x7f77f0009040 comp rx=0 tx=0).stop 2024-03-23T14:35:40.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0b2aa0 msgr2=0x7f77fc1561e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0b2aa0 0x7f77fc1561e0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f77ec00c4a0 tx=0x7f77ec00c960 comp rx=0 tx=0).stop 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 shutdown_connections 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f77e806b810 0x7f77e806dcd0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f77fc1552f0 0x7f77fc1f4bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f77fc0d0380 0x7f77fc1f3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 --2- 172.21.15.161:0/1115973060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f77fc0b2aa0 0x7f77fc1561e0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 >> 172.21.15.161:0/1115973060 conn(0x7f77fc06b260 msgr2=0x7f77fc0cedd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 shutdown_connections 2024-03-23T14:35:40.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:40.968+0000 7f7802377640 1 -- 172.21.15.161:0/1115973060 wait complete. 2024-03-23T14:35:40.975 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:41.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.083+0000 7f78737a1640 1 Processor -- start 2024-03-23T14:35:41.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.083+0000 7f78737a1640 1 -- start start 2024-03-23T14:35:41.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f78737a1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f786c15de70 0x7f786c15e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f78737a1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f786c15e790 0x7f786c0d1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f78737a1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f786c0b3710 con 0x7f786c0ce5e0 2024-03-23T14:35:41.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f78737a1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f786c15ecd0 con 0x7f786c15e790 2024-03-23T14:35:41.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f78737a1640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f786c15efb0 con 0x7f786c15de70 2024-03-23T14:35:41.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f7870d3c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f786c15e790 0x7f786c0d1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f7870d3c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f786c15e790 0x7f786c0d1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41376/0 (socket says 172.21.15.161:41376) 2024-03-23T14:35:41.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f7870d3c640 1 -- 172.21.15.161:0/3690762020 learned_addr learned my addr 172.21.15.161:0/3690762020 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:41.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.084+0000 7f787153d640 1 --2- 172.21.15.161:0/3690762020 >> v2:172.21.15.183:3300/0 conn(0x7f786c15de70 0x7f786c15e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.085+0000 7f7870d3c640 1 -- 172.21.15.161:0/3690762020 >> v2:172.21.15.183:3300/0 conn(0x7f786c15de70 msgr2=0x7f786c15e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:41.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.085+0000 7f7870d3c640 1 --2- 172.21.15.161:0/3690762020 >> v2:172.21.15.183:3300/0 conn(0x7f786c15de70 0x7f786c15e250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.085+0000 7f7870d3c640 1 -- 172.21.15.161:0/3690762020 >> v1:172.21.15.183:6789/0 conn(0x7f786c0ce5e0 legacy=0x7f786c161be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.085+0000 7f7870d3c640 1 -- 172.21.15.161:0/3690762020 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f786c15f290 con 0x7f786c15e790 2024-03-23T14:35:41.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.085+0000 7f7870d3c640 1 --2- 172.21.15.161:0/3690762020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15e790 0x7f786c0d1a20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f786c0cf820 tx=0x7f785c02b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8221e70a770937ea server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f7863fff640 1 -- 172.21.15.161:0/3690762020 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c030860 con 0x7f786c15e790 2024-03-23T14:35:41.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f7863fff640 1 -- 172.21.15.161:0/3690762020 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785c0309c0 con 0x7f786c15e790 2024-03-23T14:35:41.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f7863fff640 1 -- 172.21.15.161:0/3690762020 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c030d50 con 0x7f786c15e790 2024-03-23T14:35:41.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f78737a1640 1 -- 172.21.15.161:0/3690762020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15e790 msgr2=0x7f786c0d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f78737a1640 1 --2- 172.21.15.161:0/3690762020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15e790 0x7f786c0d1a20 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f786c0cf820 tx=0x7f785c02b9a0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f78737a1640 1 -- 172.21.15.161:0/3690762020 shutdown_connections 2024-03-23T14:35:41.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f78737a1640 1 --2- 172.21.15.161:0/3690762020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15e790 0x7f786c0d1a20 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f78737a1640 1 --2- 172.21.15.161:0/3690762020 >> v2:172.21.15.183:3300/0 conn(0x7f786c15de70 0x7f786c15e250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f78737a1640 1 -- 172.21.15.161:0/3690762020 >> 172.21.15.161:0/3690762020 conn(0x7f786c09aa40 msgr2=0x7f786c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:41.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f78737a1640 1 -- 172.21.15.161:0/3690762020 shutdown_connections 2024-03-23T14:35:41.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.086+0000 7f78737a1640 1 -- 172.21.15.161:0/3690762020 wait complete. 2024-03-23T14:35:41.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.087+0000 7f78737a1640 1 Processor -- start 2024-03-23T14:35:41.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.087+0000 7f78737a1640 1 -- start start 2024-03-23T14:35:41.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f78737a1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f786c0ce5e0 0x7f786c162010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f78737a1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15de70 0x7f786c15b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f78737a1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f786c15e790 0x7f786c0b5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f78737a1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f786c0b5bd0 con 0x7f786c15de70 2024-03-23T14:35:41.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f78737a1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f786c0b4f50 con 0x7f786c0ce5e0 2024-03-23T14:35:41.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f78737a1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f786c0b5230 con 0x7f786c15e790 2024-03-23T14:35:41.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f7870d3c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15de70 0x7f786c15b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f7870d3c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15de70 0x7f786c15b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41382/0 (socket says 172.21.15.161:41382) 2024-03-23T14:35:41.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f7870d3c640 1 -- 172.21.15.161:0/2100355867 learned_addr learned my addr 172.21.15.161:0/2100355867 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:41.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f787153d640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f786c0ce5e0 0x7f786c162010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.088+0000 7f7871d3e640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f786c15e790 0x7f786c0b5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7870d3c640 1 -- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f786c15e790 msgr2=0x7f786c0b5ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7870d3c640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f786c15e790 0x7f786c0b5ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7870d3c640 1 -- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f786c0ce5e0 msgr2=0x7f786c162010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7870d3c640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f786c0ce5e0 0x7f786c162010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7870d3c640 1 -- 172.21.15.161:0/2100355867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f785c0094d0 con 0x7f786c15de70 2024-03-23T14:35:41.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7870d3c640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15de70 0x7f786c15b290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f785c02b9e0 tx=0x7f785c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7861ffb640 1 -- 172.21.15.161:0/2100355867 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c03e040 con 0x7f786c15de70 2024-03-23T14:35:41.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7861ffb640 1 -- 172.21.15.161:0/2100355867 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f785c039c30 con 0x7f786c15de70 2024-03-23T14:35:41.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f786c0b5510 con 0x7f786c15de70 2024-03-23T14:35:41.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f7861ffb640 1 -- 172.21.15.161:0/2100355867 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f785c0443f0 con 0x7f786c15de70 2024-03-23T14:35:41.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.089+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f786c15b7d0 con 0x7f786c15de70 2024-03-23T14:35:41.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.090+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f786c09ff90 con 0x7f786c15de70 2024-03-23T14:35:41.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.090+0000 7f7861ffb640 1 -- 172.21.15.161:0/2100355867 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f785c04d3b0 con 0x7f786c15de70 2024-03-23T14:35:41.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.091+0000 7f7861ffb640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f785006b810 0x7f785006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.091+0000 7f7861ffb640 1 -- 172.21.15.161:0/2100355867 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f785c0ac880 con 0x7f786c15de70 2024-03-23T14:35:41.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.091+0000 7f787153d640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f785006b810 0x7f785006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.092+0000 7f787153d640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f785006b810 0x7f785006dcd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f78680113e0 tx=0x7f786800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.095+0000 7f7861ffb640 1 -- 172.21.15.161:0/2100355867 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f785c0773e0 con 0x7f786c15de70 2024-03-23T14:35:41.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.267+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f786c0b10f0 con 0x7f786c15de70 2024-03-23T14:35:41.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.267+0000 7f7861ffb640 1 -- 172.21.15.161:0/2100355867 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f785c07aaa0 con 0x7f786c15de70 2024-03-23T14:35:41.263 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:41.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f785006b810 msgr2=0x7f785006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f785006b810 0x7f785006dcd0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f78680113e0 tx=0x7f786800b040 comp rx=0 tx=0).stop 2024-03-23T14:35:41.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15de70 msgr2=0x7f786c15b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15de70 0x7f786c15b290 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f785c02b9e0 tx=0x7f785c004730 comp rx=0 tx=0).stop 2024-03-23T14:35:41.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 shutdown_connections 2024-03-23T14:35:41.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f785006b810 0x7f785006dcd0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f786c15e790 0x7f786c0b5ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f786c15de70 0x7f786c15b290 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 --2- 172.21.15.161:0/2100355867 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f786c0ce5e0 0x7f786c162010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 >> 172.21.15.161:0/2100355867 conn(0x7f786c09aa40 msgr2=0x7f786c0cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:41.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 shutdown_connections 2024-03-23T14:35:41.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.269+0000 7f78737a1640 1 -- 172.21.15.161:0/2100355867 wait complete. 2024-03-23T14:35:41.276 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:41.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.384+0000 7f0c17319640 1 Processor -- start 2024-03-23T14:35:41.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.384+0000 7f0c17319640 1 -- start start 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c17319640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0c1015e080 0x7f0c100ce930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c17319640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0c100cb890 0x7f0c100aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c17319640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0c10161d50 con 0x7f0c100afaf0 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c17319640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0c100cee70 con 0x7f0c1015e080 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c17319640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f0c10161b40 con 0x7f0c100cb890 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c150b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0c1015e080 0x7f0c100ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c148b4640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0c100afaf0 0x7f0c100cb180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60928/0 (socket says 172.21.15.161:60928) 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c148b4640 1 -- 172.21.15.161:0/4249928862 learned_addr learned my addr 172.21.15.161:0/4249928862 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c158b6640 1 --2- 172.21.15.161:0/4249928862 >> v2:172.21.15.183:3300/0 conn(0x7f0c100cb890 0x7f0c100aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c07fff640 1 -- 172.21.15.161:0/4249928862 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1519562261 0 0) 0x7f0c10161d50 con 0x7f0c100afaf0 2024-03-23T14:35:41.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c07fff640 1 -- 172.21.15.161:0/4249928862 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bfc003520 con 0x7f0c100afaf0 2024-03-23T14:35:41.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c150b5640 1 -- 172.21.15.161:0/4249928862 >> v2:172.21.15.183:3300/0 conn(0x7f0c100cb890 msgr2=0x7f0c100aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:41.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c150b5640 1 --2- 172.21.15.161:0/4249928862 >> v2:172.21.15.183:3300/0 conn(0x7f0c100cb890 0x7f0c100aff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c150b5640 1 -- 172.21.15.161:0/4249928862 >> v1:172.21.15.161:6789/0 conn(0x7f0c100afaf0 legacy=0x7f0c100cb180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.385+0000 7f0c150b5640 1 -- 172.21.15.161:0/4249928862 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c10162030 con 0x7f0c1015e080 2024-03-23T14:35:41.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.386+0000 7f0c150b5640 1 --2- 172.21.15.161:0/4249928862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c1015e080 0x7f0c100ce930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0c100cb130 tx=0x7f0c0c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f9f202d9193b06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.386+0000 7f0c07fff640 1 -- 172.21.15.161:0/4249928862 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c031830 con 0x7f0c1015e080 2024-03-23T14:35:41.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.386+0000 7f0c07fff640 1 -- 172.21.15.161:0/4249928862 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c0c031990 con 0x7f0c1015e080 2024-03-23T14:35:41.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.386+0000 7f0c07fff640 1 -- 172.21.15.161:0/4249928862 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c0c031d20 con 0x7f0c1015e080 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.387+0000 7f0c17319640 1 -- 172.21.15.161:0/4249928862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c1015e080 msgr2=0x7f0c100ce930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.387+0000 7f0c17319640 1 --2- 172.21.15.161:0/4249928862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c1015e080 0x7f0c100ce930 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f0c100cb130 tx=0x7f0c0c02b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.387+0000 7f0c17319640 1 -- 172.21.15.161:0/4249928862 shutdown_connections 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.387+0000 7f0c17319640 1 --2- 172.21.15.161:0/4249928862 >> v2:172.21.15.183:3300/0 conn(0x7f0c100cb890 0x7f0c100aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.387+0000 7f0c17319640 1 --2- 172.21.15.161:0/4249928862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c1015e080 0x7f0c100ce930 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.387+0000 7f0c17319640 1 -- 172.21.15.161:0/4249928862 >> 172.21.15.161:0/4249928862 conn(0x7f0c100d0b30 msgr2=0x7f0c100d0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.387+0000 7f0c17319640 1 -- 172.21.15.161:0/4249928862 shutdown_connections 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.387+0000 7f0c17319640 1 -- 172.21.15.161:0/4249928862 wait complete. 2024-03-23T14:35:41.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.388+0000 7f0c17319640 1 Processor -- start 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.388+0000 7f0c17319640 1 -- start start 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.388+0000 7f0c17319640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c100afaf0 0x7f0c10150190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.388+0000 7f0c17319640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c100cb890 0x7f0c100ca4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c17319640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c1015e080 0x7f0c100c85e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c17319640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0c100cb110 con 0x7f0c100cb890 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c17319640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0c100cae30 con 0x7f0c1015e080 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c17319640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0c1015b320 con 0x7f0c100afaf0 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c100cb890 0x7f0c100ca4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c100cb890 0x7f0c100ca4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41400/0 (socket says 172.21.15.161:41400) 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c150b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c100afaf0 0x7f0c10150190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 -- 172.21.15.161:0/1740889700 learned_addr learned my addr 172.21.15.161:0/1740889700 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c158b6640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c1015e080 0x7f0c100c85e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 -- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c100afaf0 msgr2=0x7f0c10150190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c100afaf0 0x7f0c10150190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 -- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c1015e080 msgr2=0x7f0c100c85e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c1015e080 0x7f0c100c85e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 -- 172.21.15.161:0/1740889700 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0c0c009400 con 0x7f0c100cb890 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c148b4640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c100cb890 0x7f0c100ca4e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000c430 tx=0x7f0c0000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c05ffb640 1 -- 172.21.15.161:0/1740889700 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c00005ce0 con 0x7f0c100cb890 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c05ffb640 1 -- 172.21.15.161:0/1740889700 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0c00005e40 con 0x7f0c100cb890 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0c1015b600 con 0x7f0c100cb890 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c05ffb640 1 -- 172.21.15.161:0/1740889700 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c000072f0 con 0x7f0c100cb890 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.389+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0c101506d0 con 0x7f0c100cb890 2024-03-23T14:35:41.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.390+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0c1015b180 con 0x7f0c100cb890 2024-03-23T14:35:41.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.391+0000 7f0c05ffb640 1 -- 172.21.15.161:0/1740889700 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0c00007450 con 0x7f0c100cb890 2024-03-23T14:35:41.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.391+0000 7f0c05ffb640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0bf406b810 0x7f0bf406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.391+0000 7f0c05ffb640 1 -- 172.21.15.161:0/1740889700 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0c00089790 con 0x7f0c100cb890 2024-03-23T14:35:41.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.391+0000 7f0c150b5640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0bf406b810 0x7f0bf406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.392+0000 7f0c150b5640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0bf406b810 0x7f0bf406dcd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c036000 tx=0x7f0c0c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.395+0000 7f0c05ffb640 1 -- 172.21.15.161:0/1740889700 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0c1015b180 con 0x7f0c100cb890 2024-03-23T14:35:41.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.567+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0c1015bb20 con 0x7f0c100cb890 2024-03-23T14:35:41.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.567+0000 7f0c05ffb640 1 -- 172.21.15.161:0/1740889700 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f0c1015bb20 con 0x7f0c100cb890 2024-03-23T14:35:41.563 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0bf406b810 msgr2=0x7f0bf406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0bf406b810 0x7f0bf406dcd0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0c0c036000 tx=0x7f0c0c03c040 comp rx=0 tx=0).stop 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c100cb890 msgr2=0x7f0c100ca4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c100cb890 0x7f0c100ca4e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0c0000c430 tx=0x7f0c0000c8f0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 shutdown_connections 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0bf406b810 0x7f0bf406dcd0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0c1015e080 0x7f0c100c85e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0c100cb890 0x7f0c100ca4e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 --2- 172.21.15.161:0/1740889700 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0c100afaf0 0x7f0c10150190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 >> 172.21.15.161:0/1740889700 conn(0x7f0c100d0b30 msgr2=0x7f0c100d1120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 shutdown_connections 2024-03-23T14:35:41.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.570+0000 7f0c17319640 1 -- 172.21.15.161:0/1740889700 wait complete. 2024-03-23T14:35:41.577 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:35:41.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.683+0000 7f1e6516b640 1 Processor -- start 2024-03-23T14:35:41.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.683+0000 7f1e6516b640 1 -- start start 2024-03-23T14:35:41.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.683+0000 7f1e6516b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1e60159b50 0x7f1e6015cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.683+0000 7f1e6516b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e60159f30 con 0x7f1e600d0320 2024-03-23T14:35:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.683+0000 7f1e6516b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e60071630 con 0x7f1e60071ac0 2024-03-23T14:35:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.683+0000 7f1e6516b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1e6015d470 con 0x7f1e60159b50 2024-03-23T14:35:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.684+0000 7f1e5f59e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1e600d0320 0x7f1e600ce430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60934/0 (socket says 172.21.15.161:60934) 2024-03-23T14:35:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.684+0000 7f1e5f59e640 1 -- 172.21.15.161:0/465345995 learned_addr learned my addr 172.21.15.161:0/465345995 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.684+0000 7f1e5ed9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1e60159b50 0x7f1e6015cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.684+0000 7f1e5ed9d640 1 -- 172.21.15.161:0/465345995 >> v1:172.21.15.183:6789/0 conn(0x7f1e60071ac0 legacy=0x7f1e600cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.680 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.684+0000 7f1e5ed9d640 1 -- 172.21.15.161:0/465345995 >> v1:172.21.15.161:6789/0 conn(0x7f1e600d0320 legacy=0x7f1e600ce430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.684+0000 7f1e5ed9d640 1 -- 172.21.15.161:0/465345995 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e60071020 con 0x7f1e60159b50 2024-03-23T14:35:41.681 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e5ed9d640 1 --2- 172.21.15.161:0/465345995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e60159b50 0x7f1e6015cf30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1e580097d0 tx=0x7f1e5802b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c390b55594a2844e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e5dd9b640 1 -- 172.21.15.161:0/465345995 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e5802d040 con 0x7f1e60159b50 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e5dd9b640 1 -- 172.21.15.161:0/465345995 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e58031810 con 0x7f1e60159b50 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e5dd9b640 1 -- 172.21.15.161:0/465345995 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e58031ba0 con 0x7f1e60159b50 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e6516b640 1 -- 172.21.15.161:0/465345995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e60159b50 msgr2=0x7f1e6015cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e6516b640 1 --2- 172.21.15.161:0/465345995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e60159b50 0x7f1e6015cf30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f1e580097d0 tx=0x7f1e5802b8b0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e6516b640 1 -- 172.21.15.161:0/465345995 shutdown_connections 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e6516b640 1 --2- 172.21.15.161:0/465345995 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e60159b50 0x7f1e6015cf30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e6516b640 1 -- 172.21.15.161:0/465345995 >> 172.21.15.161:0/465345995 conn(0x7f1e6006d0e0 msgr2=0x7f1e6006d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e6516b640 1 -- 172.21.15.161:0/465345995 shutdown_connections 2024-03-23T14:35:41.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.685+0000 7f1e6516b640 1 -- 172.21.15.161:0/465345995 wait complete. 2024-03-23T14:35:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.687+0000 7f1e6516b640 1 Processor -- start 2024-03-23T14:35:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.687+0000 7f1e6516b640 1 -- start start 2024-03-23T14:35:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.687+0000 7f1e6516b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e60071ac0 0x7f1e601e7010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.687+0000 7f1e6516b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e600d0320 0x7f1e60146db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.687+0000 7f1e6516b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e60159b50 0x7f1e601454e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.687+0000 7f1e6516b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1e60145250 con 0x7f1e600d0320 2024-03-23T14:35:41.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.687+0000 7f1e6516b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1e60146920 con 0x7f1e60159b50 2024-03-23T14:35:41.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.687+0000 7f1e6516b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1e601472f0 con 0x7f1e60071ac0 2024-03-23T14:35:41.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e600d0320 0x7f1e60146db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5ed9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e60071ac0 0x7f1e601e7010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e600d0320 0x7f1e60146db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41412/0 (socket says 172.21.15.161:41412) 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5f59e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e60159b50 0x7f1e601454e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 -- 172.21.15.161:0/2041207676 learned_addr learned my addr 172.21.15.161:0/2041207676 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 -- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e60071ac0 msgr2=0x7f1e601e7010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e60071ac0 0x7f1e601e7010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 -- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e60159b50 msgr2=0x7f1e601454e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e60159b50 0x7f1e601454e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 -- 172.21.15.161:0/2041207676 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e58009400 con 0x7f1e600d0320 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e5e59c640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e600d0320 0x7f1e60146db0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400c440 tx=0x7f1e54001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e477fe640 1 -- 172.21.15.161:0/2041207676 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e540052d0 con 0x7f1e600d0320 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e477fe640 1 -- 172.21.15.161:0/2041207676 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e54005430 con 0x7f1e600d0320 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e477fe640 1 -- 172.21.15.161:0/2041207676 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e54007260 con 0x7f1e600d0320 2024-03-23T14:35:41.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e60145d10 con 0x7f1e600d0320 2024-03-23T14:35:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.688+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e601462d0 con 0x7f1e600d0320 2024-03-23T14:35:41.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.689+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e6006e180 con 0x7f1e600d0320 2024-03-23T14:35:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.690+0000 7f1e477fe640 1 -- 172.21.15.161:0/2041207676 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1e540073c0 con 0x7f1e600d0320 2024-03-23T14:35:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.690+0000 7f1e477fe640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e4c06b810 0x7f1e4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.690+0000 7f1e477fe640 1 -- 172.21.15.161:0/2041207676 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1e54089530 con 0x7f1e600d0320 2024-03-23T14:35:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.690+0000 7f1e5ed9d640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e4c06b810 0x7f1e4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:41.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.691+0000 7f1e5ed9d640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e4c06b810 0x7f1e4c06dcd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e58036000 tx=0x7f1e5803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:41.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.694+0000 7f1e477fe640 1 -- 172.21.15.161:0/2041207676 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1e54054000 con 0x7f1e600d0320 2024-03-23T14:35:41.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.880+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f1e6004f550 con 0x7f1e600d0320 2024-03-23T14:35:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.881+0000 7f1e477fe640 1 -- 172.21.15.161:0/2041207676 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f1e540576c0 con 0x7f1e600d0320 2024-03-23T14:35:41.877 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e4c06b810 msgr2=0x7f1e4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e4c06b810 0x7f1e4c06dcd0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1e58036000 tx=0x7f1e5803c040 comp rx=0 tx=0).stop 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e600d0320 msgr2=0x7f1e60146db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e600d0320 0x7f1e60146db0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f1e5400c440 tx=0x7f1e54001e70 comp rx=0 tx=0).stop 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 shutdown_connections 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e4c06b810 0x7f1e4c06dcd0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e60159b50 0x7f1e601454e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e600d0320 0x7f1e60146db0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 --2- 172.21.15.161:0/2041207676 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e60071ac0 0x7f1e601e7010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:41.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 >> 172.21.15.161:0/2041207676 conn(0x7f1e6006d0e0 msgr2=0x7f1e60066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:41.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 shutdown_connections 2024-03-23T14:35:41.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:41.883+0000 7f1e6516b640 1 -- 172.21.15.161:0/2041207676 wait complete. 2024-03-23T14:35:41.890 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:35:41.890 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:41.905 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:42.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.050+0000 7f57f2623640 1 Processor -- start 2024-03-23T14:35:42.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.050+0000 7f57f2623640 1 -- start start 2024-03-23T14:35:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f2623640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f57ec06eca0 0x7f57ec165d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f2623640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57ec15be70 con 0x7f57ec072100 2024-03-23T14:35:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f2623640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57ec15c150 con 0x7f57ec0724e0 2024-03-23T14:35:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f2623640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f57ec15c430 con 0x7f57ec06eca0 2024-03-23T14:35:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f0bc0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f57ec06eca0 0x7f57ec165d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f0bc0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f57ec06eca0 0x7f57ec165d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41428/0 (socket says 172.21.15.161:41428) 2024-03-23T14:35:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57ebfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f57ec072100 0x7f57ec15b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60946/0 (socket says 172.21.15.161:60946) 2024-03-23T14:35:42.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f0bc0640 1 -- 172.21.15.161:0/3756369057 learned_addr learned my addr 172.21.15.161:0/3756369057 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f0bc0640 1 -- 172.21.15.161:0/3756369057 >> v1:172.21.15.183:6789/0 conn(0x7f57ec0724e0 legacy=0x7f57ec06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f0bc0640 1 -- 172.21.15.161:0/3756369057 >> v1:172.21.15.161:6789/0 conn(0x7f57ec072100 legacy=0x7f57ec15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57f0bc0640 1 -- 172.21.15.161:0/3756369057 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57ec0b2c70 con 0x7f57ec06eca0 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.051+0000 7f57eaffd640 1 -- 172.21.15.161:0/3756369057 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1007875976 0 0) 0x7f57ec15be70 con 0x7f57ec072100 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.052+0000 7f57f0bc0640 1 --2- 172.21.15.161:0/3756369057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec165d70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009e50 tx=0x7f57e402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=713099cb96c153ac server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.052+0000 7f57eaffd640 1 -- 172.21.15.161:0/3756369057 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e402cb20 con 0x7f57ec06eca0 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.052+0000 7f57eaffd640 1 -- 172.21.15.161:0/3756369057 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57e402cc80 con 0x7f57ec06eca0 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.052+0000 7f57eaffd640 1 -- 172.21.15.161:0/3756369057 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e40343f0 con 0x7f57ec06eca0 2024-03-23T14:35:42.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.053+0000 7f57f2623640 1 -- 172.21.15.161:0/3756369057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 msgr2=0x7f57ec165d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.053+0000 7f57f2623640 1 --2- 172.21.15.161:0/3756369057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec165d70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f57e4009e50 tx=0x7f57e402bf80 comp rx=0 tx=0).stop 2024-03-23T14:35:42.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.053+0000 7f57f2623640 1 -- 172.21.15.161:0/3756369057 shutdown_connections 2024-03-23T14:35:42.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.053+0000 7f57f2623640 1 --2- 172.21.15.161:0/3756369057 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec165d70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.053+0000 7f57f2623640 1 -- 172.21.15.161:0/3756369057 >> 172.21.15.161:0/3756369057 conn(0x7f57ec0b17f0 msgr2=0x7f57ec0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:42.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.053+0000 7f57f2623640 1 -- 172.21.15.161:0/3756369057 shutdown_connections 2024-03-23T14:35:42.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.053+0000 7f57f2623640 1 -- 172.21.15.161:0/3756369057 wait complete. 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.054+0000 7f57f2623640 1 Processor -- start 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.054+0000 7f57f2623640 1 -- start start 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57f2623640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec1f5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57f2623640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57ec072100 0x7f57ec151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57f2623640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57ec0724e0 0x7f57ec1589e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57f2623640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f57ec1521a0 con 0x7f57ec06eca0 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57f2623640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f57ec1523d0 con 0x7f57ec072100 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57f2623640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f57ec1526b0 con 0x7f57ec0724e0 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec1f5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec1f5a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41444/0 (socket says 172.21.15.161:41444) 2024-03-23T14:35:42.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 -- 172.21.15.161:0/186245878 learned_addr learned my addr 172.21.15.161:0/186245878 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57f0bc0640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57ec0724e0 0x7f57ec1589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57eb7fe640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57ec072100 0x7f57ec151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 -- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57ec0724e0 msgr2=0x7f57ec1589e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57ec0724e0 0x7f57ec1589e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 -- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57ec072100 msgr2=0x7f57ec151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57ec072100 0x7f57ec151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 -- 172.21.15.161:0/186245878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e4009b30 con 0x7f57ec06eca0 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57eb7fe640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57ec072100 0x7f57ec151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57ebfff640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec1f5a10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c460 tx=0x7f57dc00c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.055+0000 7f57e8ff9640 1 -- 172.21.15.161:0/186245878 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc007480 con 0x7f57ec06eca0 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.056+0000 7f57e8ff9640 1 -- 172.21.15.161:0/186245878 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57dc007600 con 0x7f57ec06eca0 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.056+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57ec152990 con 0x7f57ec06eca0 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.056+0000 7f57e8ff9640 1 -- 172.21.15.161:0/186245878 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc004a20 con 0x7f57ec06eca0 2024-03-23T14:35:42.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.056+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57ec142870 con 0x7f57ec06eca0 2024-03-23T14:35:42.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.056+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57ec1f2ae0 con 0x7f57ec06eca0 2024-03-23T14:35:42.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.057+0000 7f57e8ff9640 1 -- 172.21.15.161:0/186245878 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f57dc004b80 con 0x7f57ec06eca0 2024-03-23T14:35:42.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.057+0000 7f57e8ff9640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f57d806b810 0x7f57d806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.057+0000 7f57e8ff9640 1 -- 172.21.15.161:0/186245878 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f57dc0896b0 con 0x7f57ec06eca0 2024-03-23T14:35:42.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.057+0000 7f57eb7fe640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f57d806b810 0x7f57d806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.058+0000 7f57eb7fe640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f57d806b810 0x7f57d806dcd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f57e0007fd0 tx=0x7f57e0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.061+0000 7f57e8ff9640 1 -- 172.21.15.161:0/186245878 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f57ec1f2ae0 con 0x7f57ec06eca0 2024-03-23T14:35:42.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.233+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f57ec1f2ec0 con 0x7f57ec06eca0 2024-03-23T14:35:42.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.234+0000 7f57e8ff9640 1 -- 172.21.15.161:0/186245878 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f57ec1f2ec0 con 0x7f57ec06eca0 2024-03-23T14:35:42.230 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:42.230 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:42.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f57d806b810 msgr2=0x7f57d806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f57d806b810 0x7f57d806dcd0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f57e0007fd0 tx=0x7f57e0009040 comp rx=0 tx=0).stop 2024-03-23T14:35:42.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 msgr2=0x7f57ec1f5a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec1f5a10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00c460 tx=0x7f57dc00c920 comp rx=0 tx=0).stop 2024-03-23T14:35:42.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 shutdown_connections 2024-03-23T14:35:42.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f57d806b810 0x7f57d806dcd0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f57ec0724e0 0x7f57ec1589e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f57ec072100 0x7f57ec151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 --2- 172.21.15.161:0/186245878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f57ec06eca0 0x7f57ec1f5a10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 >> 172.21.15.161:0/186245878 conn(0x7f57ec0b17f0 msgr2=0x7f57ec0cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:42.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 shutdown_connections 2024-03-23T14:35:42.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.236+0000 7f57f2623640 1 -- 172.21.15.161:0/186245878 wait complete. 2024-03-23T14:35:42.243 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:42.244 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:42.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.352+0000 7fd03b593640 1 Processor -- start 2024-03-23T14:35:42.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.352+0000 7fd03b593640 1 -- start start 2024-03-23T14:35:42.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03b593640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0341552f0 0x7fd0340b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03b593640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0340b2aa0 0x7fd0340cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03b593640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd03415f480 con 0x7fd0340b4250 2024-03-23T14:35:42.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03b593640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd0341556d0 con 0x7fd0341552f0 2024-03-23T14:35:42.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03b593640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd03406eee0 con 0x7fd0340b2aa0 2024-03-23T14:35:42.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03932f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0341552f0 0x7fd0340b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03932f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd0341552f0 0x7fd0340b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41446/0 (socket says 172.21.15.161:41446) 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03932f640 1 -- 172.21.15.161:0/4260849363 learned_addr learned my addr 172.21.15.161:0/4260849363 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd038b2e640 1 --2- 172.21.15.161:0/4260849363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0340b2aa0 0x7fd0340cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03932f640 1 -- 172.21.15.161:0/4260849363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0340b2aa0 msgr2=0x7fd0340cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.353+0000 7fd03932f640 1 --2- 172.21.15.161:0/4260849363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0340b2aa0 0x7fd0340cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.354+0000 7fd03932f640 1 -- 172.21.15.161:0/4260849363 >> v1:172.21.15.183:6789/0 conn(0x7fd0340b4250 legacy=0x7fd034071c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.354+0000 7fd03932f640 1 -- 172.21.15.161:0/4260849363 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03415e870 con 0x7fd0341552f0 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.354+0000 7fd03932f640 1 --2- 172.21.15.161:0/4260849363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0341552f0 0x7fd0340b3d10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd03000d150 tx=0x7fd03002e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e09b4fe8e6cccb3b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.354+0000 7fd02bfff640 1 -- 172.21.15.161:0/4260849363 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03000b0d0 con 0x7fd0341552f0 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.354+0000 7fd02bfff640 1 -- 172.21.15.161:0/4260849363 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd03000b230 con 0x7fd0341552f0 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.354+0000 7fd02bfff640 1 -- 172.21.15.161:0/4260849363 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd030009600 con 0x7fd0341552f0 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.355+0000 7fd03b593640 1 -- 172.21.15.161:0/4260849363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0341552f0 msgr2=0x7fd0340b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.355+0000 7fd03b593640 1 --2- 172.21.15.161:0/4260849363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0341552f0 0x7fd0340b3d10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd03000d150 tx=0x7fd03002e160 comp rx=0 tx=0).stop 2024-03-23T14:35:42.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.355+0000 7fd03b593640 1 -- 172.21.15.161:0/4260849363 shutdown_connections 2024-03-23T14:35:42.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.355+0000 7fd03b593640 1 --2- 172.21.15.161:0/4260849363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0340b2aa0 0x7fd0340cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.355+0000 7fd03b593640 1 --2- 172.21.15.161:0/4260849363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0341552f0 0x7fd0340b3d10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.355+0000 7fd03b593640 1 -- 172.21.15.161:0/4260849363 >> 172.21.15.161:0/4260849363 conn(0x7fd03406b260 msgr2=0x7fd03406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:42.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.355+0000 7fd03b593640 1 -- 172.21.15.161:0/4260849363 shutdown_connections 2024-03-23T14:35:42.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.355+0000 7fd03b593640 1 -- 172.21.15.161:0/4260849363 wait complete. 2024-03-23T14:35:42.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.356+0000 7fd03b593640 1 Processor -- start 2024-03-23T14:35:42.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.356+0000 7fd03b593640 1 -- start start 2024-03-23T14:35:42.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd03b593640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0340b2aa0 0x7fd034070b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd03b593640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0340b4250 0x7fd03415a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd03b593640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0341552f0 0x7fd034145bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd03b593640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd034159110 con 0x7fd0340b4250 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd03b593640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd034158ae0 con 0x7fd0340b2aa0 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd03b593640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd0341468e0 con 0x7fd0341552f0 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd038b2e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0340b4250 0x7fd03415a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd038b2e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0340b4250 0x7fd03415a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41458/0 (socket says 172.21.15.161:41458) 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd038b2e640 1 -- 172.21.15.161:0/998321844 learned_addr learned my addr 172.21.15.161:0/998321844 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd039b30640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0341552f0 0x7fd034145bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd03932f640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0340b2aa0 0x7fd034070b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd038b2e640 1 -- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0341552f0 msgr2=0x7fd034145bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd038b2e640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0341552f0 0x7fd034145bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd038b2e640 1 -- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0340b2aa0 msgr2=0x7fd034070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd038b2e640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0340b2aa0 0x7fd034070b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.357+0000 7fd038b2e640 1 -- 172.21.15.161:0/998321844 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd03000bd40 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.358+0000 7fd038b2e640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0340b4250 0x7fd03415a9b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd02400e870 tx=0x7fd02400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.358+0000 7fd029ffb640 1 -- 172.21.15.161:0/998321844 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0240094f0 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.358+0000 7fd029ffb640 1 -- 172.21.15.161:0/998321844 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd02400c940 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.358+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0340b0410 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.358+0000 7fd029ffb640 1 -- 172.21.15.161:0/998321844 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02400ccf0 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.358+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd034158d30 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.359+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd034159380 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.359+0000 7fd029ffb640 1 -- 172.21.15.161:0/998321844 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd024013470 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.360+0000 7fd029ffb640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd01806b810 0x7fd01806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.360+0000 7fd029ffb640 1 -- 172.21.15.161:0/998321844 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd02408a160 con 0x7fd0340b4250 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.360+0000 7fd03932f640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd01806b810 0x7fd01806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.360+0000 7fd03932f640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd01806b810 0x7fd01806dcd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd030037000 tx=0x7fd030032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.364+0000 7fd029ffb640 1 -- 172.21.15.161:0/998321844 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd024054cc0 con 0x7fd0340b4250 2024-03-23T14:35:42.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.533+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd03404f4e0 con 0x7fd0340b4250 2024-03-23T14:35:42.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.534+0000 7fd029ffb640 1 -- 172.21.15.161:0/998321844 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fd024058380 con 0x7fd0340b4250 2024-03-23T14:35:42.529 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.535+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd01806b810 msgr2=0x7fd01806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.535+0000 7fd03b593640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd01806b810 0x7fd01806dcd0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd030037000 tx=0x7fd030032000 comp rx=0 tx=0).stop 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.535+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0340b4250 msgr2=0x7fd03415a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.535+0000 7fd03b593640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0340b4250 0x7fd03415a9b0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd02400e870 tx=0x7fd02400ed30 comp rx=0 tx=0).stop 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.536+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 shutdown_connections 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.536+0000 7fd03b593640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd01806b810 0x7fd01806dcd0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.536+0000 7fd03b593640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0341552f0 0x7fd034145bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.536+0000 7fd03b593640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd0340b4250 0x7fd03415a9b0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.536+0000 7fd03b593640 1 --2- 172.21.15.161:0/998321844 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd0340b2aa0 0x7fd034070b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.536+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 >> 172.21.15.161:0/998321844 conn(0x7fd03406b260 msgr2=0x7fd0340cfc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:42.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.536+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 shutdown_connections 2024-03-23T14:35:42.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.536+0000 7fd03b593640 1 -- 172.21.15.161:0/998321844 wait complete. 2024-03-23T14:35:42.542 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:42.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.650+0000 7f1e76ad4640 1 Processor -- start 2024-03-23T14:35:42.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.650+0000 7f1e76ad4640 1 -- start start 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e76ad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e701549c0 0x7f1e7006e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e76ad4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1e7015b780 0x7f1e7006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e76ad4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1e700b2c20 con 0x7f1e70070c20 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e76ad4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f1e7006ee90 con 0x7f1e7015b780 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e76ad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1e700b2a10 con 0x7f1e701549c0 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e6ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1e70070c20 0x7f1e7015b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60952/0 (socket says 172.21.15.161:60952) 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e6ffff640 1 -- 172.21.15.161:0/2367401787 learned_addr learned my addr 172.21.15.161:0/2367401787 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e74870640 1 --2- 172.21.15.161:0/2367401787 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e701549c0 0x7f1e7006e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e75071640 1 --2- 172.21.15.161:0/2367401787 >> v2:172.21.15.183:3300/0 conn(0x7f1e7015b780 0x7f1e7006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e6f7fe640 1 -- 172.21.15.161:0/2367401787 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3483537840 0 0) 0x7f1e700b2c20 con 0x7f1e70070c20 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e6f7fe640 1 -- 172.21.15.161:0/2367401787 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e60003520 con 0x7f1e70070c20 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e74870640 1 -- 172.21.15.161:0/2367401787 >> v2:172.21.15.183:3300/0 conn(0x7f1e7015b780 msgr2=0x7f1e7006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e74870640 1 --2- 172.21.15.161:0/2367401787 >> v2:172.21.15.183:3300/0 conn(0x7f1e7015b780 0x7f1e7006d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e74870640 1 -- 172.21.15.161:0/2367401787 >> v1:172.21.15.161:6789/0 conn(0x7f1e70070c20 legacy=0x7f1e7015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e74870640 1 -- 172.21.15.161:0/2367401787 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e700b2f00 con 0x7f1e701549c0 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.651+0000 7f1e75071640 1 --2- 172.21.15.161:0/2367401787 >> v2:172.21.15.183:3300/0 conn(0x7f1e7015b780 0x7f1e7006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.652+0000 7f1e74870640 1 --2- 172.21.15.161:0/2367401787 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e701549c0 0x7f1e7006e950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400a940 tx=0x7f1e6402cd80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7284930c7193a4d3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.652+0000 7f1e6f7fe640 1 -- 172.21.15.161:0/2367401787 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e6402db30 con 0x7f1e701549c0 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.652+0000 7f1e6f7fe640 1 -- 172.21.15.161:0/2367401787 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e6402dc90 con 0x7f1e701549c0 2024-03-23T14:35:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.652+0000 7f1e6f7fe640 1 -- 172.21.15.161:0/2367401787 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64035440 con 0x7f1e701549c0 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.653+0000 7f1e76ad4640 1 -- 172.21.15.161:0/2367401787 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e701549c0 msgr2=0x7f1e7006e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.653+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/2367401787 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e701549c0 0x7f1e7006e950 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f1e6400a940 tx=0x7f1e6402cd80 comp rx=0 tx=0).stop 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.653+0000 7f1e76ad4640 1 -- 172.21.15.161:0/2367401787 shutdown_connections 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.653+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/2367401787 >> v2:172.21.15.183:3300/0 conn(0x7f1e7015b780 0x7f1e7006d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.653+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/2367401787 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e701549c0 0x7f1e7006e950 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.653+0000 7f1e76ad4640 1 -- 172.21.15.161:0/2367401787 >> 172.21.15.161:0/2367401787 conn(0x7f1e7006b4a0 msgr2=0x7f1e7006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.653+0000 7f1e76ad4640 1 -- 172.21.15.161:0/2367401787 shutdown_connections 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.653+0000 7f1e76ad4640 1 -- 172.21.15.161:0/2367401787 wait complete. 2024-03-23T14:35:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.654+0000 7f1e76ad4640 1 Processor -- start 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.654+0000 7f1e76ad4640 1 -- start start 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e76ad4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e70070c20 0x7f1e70164c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e76ad4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e701549c0 0x7f1e7014eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e76ad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e7015b780 0x7f1e70152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e76ad4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1e7014f040 con 0x7f1e70070c20 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e76ad4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1e701532d0 con 0x7f1e701549c0 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e76ad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1e7014daa0 con 0x7f1e7015b780 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e70070c20 0x7f1e70164c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e70070c20 0x7f1e70164c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41470/0 (socket says 172.21.15.161:41470) 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 -- 172.21.15.161:0/3789055501 learned_addr learned my addr 172.21.15.161:0/3789055501 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e75071640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e7015b780 0x7f1e70152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e6ffff640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e701549c0 0x7f1e7014eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 -- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e7015b780 msgr2=0x7f1e70152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e7015b780 0x7f1e70152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 -- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e701549c0 msgr2=0x7f1e7014eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e701549c0 0x7f1e7014eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 -- 172.21.15.161:0/3789055501 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e68009a70 con 0x7f1e70070c20 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.655+0000 7f1e74870640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e70070c20 0x7f1e70164c30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1e64038040 tx=0x7f1e640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.656+0000 7f1e6d7fa640 1 -- 172.21.15.161:0/3789055501 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e64006e10 con 0x7f1e70070c20 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.656+0000 7f1e6d7fa640 1 -- 172.21.15.161:0/3789055501 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e64040440 con 0x7f1e70070c20 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.656+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e6400a5a0 con 0x7f1e70070c20 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.656+0000 7f1e6d7fa640 1 -- 172.21.15.161:0/3789055501 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e640407d0 con 0x7f1e70070c20 2024-03-23T14:35:42.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.656+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e7014e020 con 0x7f1e70070c20 2024-03-23T14:35:42.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.657+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e7006b880 con 0x7f1e70070c20 2024-03-23T14:35:42.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.657+0000 7f1e6d7fa640 1 -- 172.21.15.161:0/3789055501 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1e6403c040 con 0x7f1e70070c20 2024-03-23T14:35:42.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.657+0000 7f1e6d7fa640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e5406b810 0x7f1e5406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.657+0000 7f1e6d7fa640 1 -- 172.21.15.161:0/3789055501 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1e640ac1a0 con 0x7f1e70070c20 2024-03-23T14:35:42.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.658+0000 7f1e6ffff640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e5406b810 0x7f1e5406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.658+0000 7f1e6ffff640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e5406b810 0x7f1e5406dcd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1e7015d8c0 tx=0x7f1e5c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.662+0000 7f1e6d7fa640 1 -- 172.21.15.161:0/3789055501 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1e64076d70 con 0x7f1e70070c20 2024-03-23T14:35:42.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.834+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1e70073570 con 0x7f1e70070c20 2024-03-23T14:35:42.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.834+0000 7f1e6d7fa640 1 -- 172.21.15.161:0/3789055501 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f1e6407a430 con 0x7f1e70070c20 2024-03-23T14:35:42.831 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:42.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.836+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e5406b810 msgr2=0x7f1e5406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.836+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e5406b810 0x7f1e5406dcd0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1e7015d8c0 tx=0x7f1e5c009040 comp rx=0 tx=0).stop 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.836+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e70070c20 msgr2=0x7f1e70164c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.836+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e70070c20 0x7f1e70164c30 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f1e64038040 tx=0x7f1e640046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.837+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 shutdown_connections 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.837+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e5406b810 0x7f1e5406dcd0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.837+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1e7015b780 0x7f1e70152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.837+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1e701549c0 0x7f1e7014eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.837+0000 7f1e76ad4640 1 --2- 172.21.15.161:0/3789055501 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1e70070c20 0x7f1e70164c30 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.837+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 >> 172.21.15.161:0/3789055501 conn(0x7f1e7006b4a0 msgr2=0x7f1e7015dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:42.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.837+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 shutdown_connections 2024-03-23T14:35:42.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.837+0000 7f1e76ad4640 1 -- 172.21.15.161:0/3789055501 wait complete. 2024-03-23T14:35:42.844 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:42.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.951+0000 7f1fcff13640 1 Processor -- start 2024-03-23T14:35:42.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.951+0000 7f1fcff13640 1 -- start start 2024-03-23T14:35:42.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcff13640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc81552f0 0x7f1fc80b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcff13640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc815f480 con 0x7f1fc80d0380 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcff13640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fc81556d0 con 0x7f1fc80b2aa0 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcff13640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc815e870 con 0x7f1fc81552f0 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fce4b0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1fc80d0380 0x7f1fc806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60964/0 (socket says 172.21.15.161:60964) 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fce4b0640 1 -- 172.21.15.161:0/2238999908 learned_addr learned my addr 172.21.15.161:0/2238999908 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcdcaf640 1 --2- 172.21.15.161:0/2238999908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc81552f0 0x7f1fc80b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcccad640 1 -- 172.21.15.161:0/2238999908 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 282781466 0 0) 0x7f1fc815f480 con 0x7f1fc80d0380 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcccad640 1 -- 172.21.15.161:0/2238999908 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fb4003520 con 0x7f1fc80d0380 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcccad640 1 -- 172.21.15.161:0/2238999908 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 444768694 0 0) 0x7f1fc81556d0 con 0x7f1fc80b2aa0 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcccad640 1 -- 172.21.15.161:0/2238999908 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc815f480 con 0x7f1fc80b2aa0 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcdcaf640 1 -- 172.21.15.161:0/2238999908 >> v1:172.21.15.183:6789/0 conn(0x7f1fc80b2aa0 legacy=0x7f1fc80cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcdcaf640 1 -- 172.21.15.161:0/2238999908 >> v1:172.21.15.161:6789/0 conn(0x7f1fc80d0380 legacy=0x7f1fc806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.952+0000 7f1fcdcaf640 1 -- 172.21.15.161:0/2238999908 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc815eb50 con 0x7f1fc81552f0 2024-03-23T14:35:42.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.953+0000 7f1fcdcaf640 1 --2- 172.21.15.161:0/2238999908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc81552f0 0x7f1fc80b3d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40097d0 tx=0x7f1fc402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e938c28375f545e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.953+0000 7f1fcccad640 1 -- 172.21.15.161:0/2238999908 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4031890 con 0x7f1fc81552f0 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.953+0000 7f1fcccad640 1 -- 172.21.15.161:0/2238999908 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fc40319f0 con 0x7f1fc81552f0 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.953+0000 7f1fcccad640 1 -- 172.21.15.161:0/2238999908 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fc4031d80 con 0x7f1fc81552f0 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.954+0000 7f1fcff13640 1 -- 172.21.15.161:0/2238999908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc81552f0 msgr2=0x7f1fc80b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.954+0000 7f1fcff13640 1 --2- 172.21.15.161:0/2238999908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc81552f0 0x7f1fc80b3d10 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f1fc40097d0 tx=0x7f1fc402b930 comp rx=0 tx=0).stop 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.954+0000 7f1fcff13640 1 -- 172.21.15.161:0/2238999908 shutdown_connections 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.954+0000 7f1fcff13640 1 --2- 172.21.15.161:0/2238999908 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc81552f0 0x7f1fc80b3d10 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.954+0000 7f1fcff13640 1 -- 172.21.15.161:0/2238999908 >> 172.21.15.161:0/2238999908 conn(0x7f1fc806b260 msgr2=0x7f1fc806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.954+0000 7f1fcff13640 1 -- 172.21.15.161:0/2238999908 shutdown_connections 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.954+0000 7f1fcff13640 1 -- 172.21.15.161:0/2238999908 wait complete. 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.955+0000 7f1fcff13640 1 Processor -- start 2024-03-23T14:35:42.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.955+0000 7f1fcff13640 1 -- start start 2024-03-23T14:35:42.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcff13640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fc80b2aa0 0x7f1fc8158160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcff13640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc80d0380 0x7f1fc80cd170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcff13640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc813d300 0x7f1fc80cea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcff13640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc80cd9c0 con 0x7f1fc80d0380 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcff13640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1fc80cdca0 con 0x7f1fc80b2aa0 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcff13640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1fc80cdf80 con 0x7f1fc813d300 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc80d0380 0x7f1fc80cd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc80d0380 0x7f1fc80cd170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41478/0 (socket says 172.21.15.161:41478) 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 -- 172.21.15.161:0/1743168075 learned_addr learned my addr 172.21.15.161:0/1743168075 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcdcaf640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fc80b2aa0 0x7f1fc8158160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fce4b0640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc813d300 0x7f1fc80cea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 -- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc813d300 msgr2=0x7f1fc80cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc813d300 0x7f1fc80cea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 -- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fc80b2aa0 msgr2=0x7f1fc8158160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fc80b2aa0 0x7f1fc8158160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 -- 172.21.15.161:0/1743168075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fc4009400 con 0x7f1fc80d0380 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.956+0000 7f1fcd4ae640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc80d0380 0x7f1fc80cd170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800c500 tx=0x7f1fb800c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.957+0000 7f1fbe7fc640 1 -- 172.21.15.161:0/1743168075 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb80052d0 con 0x7f1fc80d0380 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.957+0000 7f1fbe7fc640 1 -- 172.21.15.161:0/1743168075 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1fb8005450 con 0x7f1fc80d0380 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.957+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fc80ce260 con 0x7f1fc80d0380 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.957+0000 7f1fbe7fc640 1 -- 172.21.15.161:0/1743168075 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8012870 con 0x7f1fc80d0380 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.957+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fc80cd6b0 con 0x7f1fc80d0380 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.957+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc80ce780 con 0x7f1fc80d0380 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.958+0000 7f1fbe7fc640 1 -- 172.21.15.161:0/1743168075 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1fb80129d0 con 0x7f1fc80d0380 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.958+0000 7f1fbe7fc640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1fac06b810 0x7f1fac06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.958+0000 7f1fbe7fc640 1 -- 172.21.15.161:0/1743168075 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f1fb8089cc0 con 0x7f1fc80d0380 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.958+0000 7f1fcdcaf640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1fac06b810 0x7f1fac06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:42.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.959+0000 7f1fcdcaf640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1fac06b810 0x7f1fac06dcd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4036000 tx=0x7f1fc403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:42.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:42.963+0000 7f1fbe7fc640 1 -- 172.21.15.161:0/1743168075 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1fc80ce780 con 0x7f1fc80d0380 2024-03-23T14:35:43.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.133+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1fc80ce780 con 0x7f1fc80d0380 2024-03-23T14:35:43.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.134+0000 7f1fbe7fc640 1 -- 172.21.15.161:0/1743168075 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f1fc80ce780 con 0x7f1fc80d0380 2024-03-23T14:35:43.130 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.135+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1fac06b810 msgr2=0x7f1fac06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.135+0000 7f1fcff13640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1fac06b810 0x7f1fac06dcd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f1fc4036000 tx=0x7f1fc403c040 comp rx=0 tx=0).stop 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.135+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc80d0380 msgr2=0x7f1fc80cd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.135+0000 7f1fcff13640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc80d0380 0x7f1fc80cd170 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f1fb800c500 tx=0x7f1fb800c9c0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.136+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 shutdown_connections 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.136+0000 7f1fcff13640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1fac06b810 0x7f1fac06dcd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.136+0000 7f1fcff13640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1fc813d300 0x7f1fc80cea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.136+0000 7f1fcff13640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1fc80d0380 0x7f1fc80cd170 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.136+0000 7f1fcff13640 1 --2- 172.21.15.161:0/1743168075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1fc80b2aa0 0x7f1fc8158160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.136+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 >> 172.21.15.161:0/1743168075 conn(0x7f1fc806b260 msgr2=0x7f1fc806e8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.136+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 shutdown_connections 2024-03-23T14:35:43.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.136+0000 7f1fcff13640 1 -- 172.21.15.161:0/1743168075 wait complete. 2024-03-23T14:35:43.143 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:35:43.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.249+0000 7fd92fa8f640 1 Processor -- start 2024-03-23T14:35:43.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.249+0000 7fd92fa8f640 1 -- start start 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92fa8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd92815b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92fa8f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd92815be70 con 0x7fd9280724e0 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92fa8f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd92815c150 con 0x7fd92806eca0 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92fa8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd92815c430 con 0x7fd928072100 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92d02a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd9280724e0 0x7fd92806e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60970/0 (socket says 172.21.15.161:60970) 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92d82b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd92815b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92d02a640 1 -- 172.21.15.161:0/3448318953 learned_addr learned my addr 172.21.15.161:0/3448318953 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92c829640 1 -- 172.21.15.161:0/3448318953 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499417571 0 0) 0x7fd92815be70 con 0x7fd9280724e0 2024-03-23T14:35:43.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92d82b640 1 -- 172.21.15.161:0/3448318953 >> v1:172.21.15.183:6789/0 conn(0x7fd92806eca0 legacy=0x7fd9280b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92d82b640 1 -- 172.21.15.161:0/3448318953 >> v1:172.21.15.161:6789/0 conn(0x7fd9280724e0 legacy=0x7fd92806e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.250+0000 7fd92d82b640 1 -- 172.21.15.161:0/3448318953 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9280b2c70 con 0x7fd928072100 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.251+0000 7fd92d82b640 1 --2- 172.21.15.161:0/3448318953 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd92815b3f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd9240097d0 tx=0x7fd92402b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61ecbb89f2673eef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.251+0000 7fd92c829640 1 -- 172.21.15.161:0/3448318953 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924031890 con 0x7fd928072100 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.251+0000 7fd92c829640 1 -- 172.21.15.161:0/3448318953 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9240319f0 con 0x7fd928072100 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.251+0000 7fd92c829640 1 -- 172.21.15.161:0/3448318953 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924031d80 con 0x7fd928072100 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.252+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3448318953 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 msgr2=0x7fd92815b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.252+0000 7fd92fa8f640 1 --2- 172.21.15.161:0/3448318953 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd92815b3f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd9240097d0 tx=0x7fd92402b930 comp rx=0 tx=0).stop 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.252+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3448318953 shutdown_connections 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.252+0000 7fd92fa8f640 1 --2- 172.21.15.161:0/3448318953 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd92815b3f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.252+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3448318953 >> 172.21.15.161:0/3448318953 conn(0x7fd9280b17f0 msgr2=0x7fd9280b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:43.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.252+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3448318953 shutdown_connections 2024-03-23T14:35:43.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.252+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3448318953 wait complete. 2024-03-23T14:35:43.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.253+0000 7fd92fa8f640 1 Processor -- start 2024-03-23T14:35:43.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.253+0000 7fd92fa8f640 1 -- start start 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92fa8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd92806eca0 0x7fd9281f5010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92fa8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd928159960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92fa8f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9280724e0 0x7fd92814f7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92fa8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281e4f60 con 0x7fd92806eca0 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92fa8f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd9281e4c80 con 0x7fd9280724e0 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92fa8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd928151f10 con 0x7fd928072100 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd92806eca0 0x7fd9281f5010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd92806eca0 0x7fd9281f5010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41484/0 (socket says 172.21.15.161:41484) 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 -- 172.21.15.161:0/3200573882 learned_addr learned my addr 172.21.15.161:0/3200573882 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d02a640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd928159960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92e02c640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9280724e0 0x7fd92814f7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 -- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 msgr2=0x7fd928159960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd928159960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 -- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9280724e0 msgr2=0x7fd92814f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9280724e0 0x7fd92814f7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 -- 172.21.15.161:0/3200573882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd924009400 con 0x7fd92806eca0 2024-03-23T14:35:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92d82b640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd92806eca0 0x7fd9281f5010 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd9240098f0 tx=0x7fd9240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd91e7fc640 1 -- 172.21.15.161:0/3200573882 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd924031830 con 0x7fd92806eca0 2024-03-23T14:35:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd91e7fc640 1 -- 172.21.15.161:0/3200573882 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9240453f0 con 0x7fd92806eca0 2024-03-23T14:35:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd91e7fc640 1 -- 172.21.15.161:0/3200573882 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9240457a0 con 0x7fd92806eca0 2024-03-23T14:35:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.254+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9281521f0 con 0x7fd92806eca0 2024-03-23T14:35:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.255+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd928159ea0 con 0x7fd92806eca0 2024-03-23T14:35:43.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.255+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd928151d70 con 0x7fd92806eca0 2024-03-23T14:35:43.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.256+0000 7fd91e7fc640 1 -- 172.21.15.161:0/3200573882 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd92404e450 con 0x7fd92806eca0 2024-03-23T14:35:43.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.256+0000 7fd91e7fc640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd90c06b810 0x7fd90c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.256+0000 7fd91e7fc640 1 -- 172.21.15.161:0/3200573882 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd9240ac820 con 0x7fd92806eca0 2024-03-23T14:35:43.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.256+0000 7fd92d02a640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd90c06b810 0x7fd90c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.257+0000 7fd92d02a640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd90c06b810 0x7fd90c06dcd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd9281f1a40 tx=0x7fd918007e40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.260+0000 7fd91e7fc640 1 -- 172.21.15.161:0/3200573882 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd928151d70 con 0x7fd92806eca0 2024-03-23T14:35:43.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.447+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fd928151d70 con 0x7fd92806eca0 2024-03-23T14:35:43.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.447+0000 7fd91e7fc640 1 -- 172.21.15.161:0/3200573882 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7fd928151d70 con 0x7fd92806eca0 2024-03-23T14:35:43.443 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:35:43.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd90c06b810 msgr2=0x7fd90c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd90c06b810 0x7fd90c06dcd0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd9281f1a40 tx=0x7fd918007e40 comp rx=0 tx=0).stop 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd92806eca0 msgr2=0x7fd9281f5010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd92806eca0 0x7fd9281f5010 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd9240098f0 tx=0x7fd9240046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 shutdown_connections 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd90c06b810 0x7fd90c06dcd0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd9280724e0 0x7fd92814f7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd928072100 0x7fd928159960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 --2- 172.21.15.161:0/3200573882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd92806eca0 0x7fd9281f5010 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 >> 172.21.15.161:0/3200573882 conn(0x7fd9280b17f0 msgr2=0x7fd928071000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 shutdown_connections 2024-03-23T14:35:43.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.449+0000 7fd92fa8f640 1 -- 172.21.15.161:0/3200573882 wait complete. 2024-03-23T14:35:43.456 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-23T14:35:43.456 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:43.456 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T14:35:43.484 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:35:43.485 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:35:43.485 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:43.485 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T14:35:43.543 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:35:43.544 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:43.560 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:35:43.560 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:43.617 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:43.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.722+0000 7f3c1b34b640 1 Processor -- start 2024-03-23T14:35:43.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.722+0000 7f3c1b34b640 1 -- start start 2024-03-23T14:35:43.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c1b34b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c140724e0 0x7f3c1406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c1b34b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c1406ead0 0x7f3c140b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c1b34b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c1415be70 con 0x7f3c14072100 2024-03-23T14:35:43.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c1b34b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3c1406ef80 con 0x7f3c140724e0 2024-03-23T14:35:43.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c1b34b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3c1415c430 con 0x7f3c1406ead0 2024-03-23T14:35:43.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c188e6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c140724e0 0x7f3c1406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c188e6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3c140724e0 0x7f3c1406e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41498/0 (socket says 172.21.15.161:41498) 2024-03-23T14:35:43.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c188e6640 1 -- 172.21.15.161:0/1797262969 learned_addr learned my addr 172.21.15.161:0/1797262969 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:43.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c198e8640 1 --2- 172.21.15.161:0/1797262969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c1406ead0 0x7f3c140b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c188e6640 1 -- 172.21.15.161:0/1797262969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c1406ead0 msgr2=0x7f3c140b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:43.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c188e6640 1 --2- 172.21.15.161:0/1797262969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c1406ead0 0x7f3c140b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c188e6640 1 -- 172.21.15.161:0/1797262969 >> v1:172.21.15.183:6789/0 conn(0x7f3c14072100 legacy=0x7f3c1415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.723+0000 7f3c188e6640 1 -- 172.21.15.161:0/1797262969 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c1415c150 con 0x7f3c140724e0 2024-03-23T14:35:43.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.724+0000 7f3c188e6640 1 --2- 172.21.15.161:0/1797262969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c140724e0 0x7f3c1406e590 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009870 tx=0x7f3c0402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2425b87fc07d15a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:43.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.724+0000 7f3c0bfff640 1 -- 172.21.15.161:0/1797262969 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04030840 con 0x7f3c140724e0 2024-03-23T14:35:43.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.724+0000 7f3c0bfff640 1 -- 172.21.15.161:0/1797262969 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c040309a0 con 0x7f3c140724e0 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.724+0000 7f3c0bfff640 1 -- 172.21.15.161:0/1797262969 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c04030d30 con 0x7f3c140724e0 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.724+0000 7f3c1b34b640 1 -- 172.21.15.161:0/1797262969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c140724e0 msgr2=0x7f3c1406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.724+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/1797262969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c140724e0 0x7f3c1406e590 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3c04009870 tx=0x7f3c0402b980 comp rx=0 tx=0).stop 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.725+0000 7f3c1b34b640 1 -- 172.21.15.161:0/1797262969 shutdown_connections 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.725+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/1797262969 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c1406ead0 0x7f3c140b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.725+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/1797262969 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c140724e0 0x7f3c1406e590 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.725+0000 7f3c1b34b640 1 -- 172.21.15.161:0/1797262969 >> 172.21.15.161:0/1797262969 conn(0x7f3c140b17f0 msgr2=0x7f3c140b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.725+0000 7f3c1b34b640 1 -- 172.21.15.161:0/1797262969 shutdown_connections 2024-03-23T14:35:43.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.725+0000 7f3c1b34b640 1 -- 172.21.15.161:0/1797262969 wait complete. 2024-03-23T14:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.726+0000 7f3c1b34b640 1 Processor -- start 2024-03-23T14:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.726+0000 7f3c1b34b640 1 -- start start 2024-03-23T14:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.726+0000 7f3c1b34b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1406ead0 0x7f3c1414f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.726+0000 7f3c1b34b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c14072100 0x7f3c14138380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.726+0000 7f3c1b34b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c140724e0 0x7f3c14150520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.726+0000 7f3c1b34b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1414f540 con 0x7f3c1406ead0 2024-03-23T14:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.726+0000 7f3c1b34b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3c1414e930 con 0x7f3c140724e0 2024-03-23T14:35:43.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.726+0000 7f3c1b34b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3c1414ec10 con 0x7f3c14072100 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1406ead0 0x7f3c1414f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1406ead0 0x7f3c1414f9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41514/0 (socket says 172.21.15.161:41514) 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 -- 172.21.15.161:0/2538266631 learned_addr learned my addr 172.21.15.161:0/2538266631 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c188e6640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c14072100 0x7f3c14138380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c198e8640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c140724e0 0x7f3c14150520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 -- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c14072100 msgr2=0x7f3c14138380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c14072100 0x7f3c14138380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 -- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c140724e0 msgr2=0x7f3c14150520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c140724e0 0x7f3c14150520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 -- 172.21.15.161:0/2538266631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c040094d0 con 0x7f3c1406ead0 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c190e7640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1406ead0 0x7f3c1414f9d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000c400 tx=0x7f3c1000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c09ffb640 1 -- 172.21.15.161:0/2538266631 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c100052d0 con 0x7f3c1406ead0 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c09ffb640 1 -- 172.21.15.161:0/2538266631 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c10005450 con 0x7f3c1406ead0 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c1414eef0 con 0x7f3c1406ead0 2024-03-23T14:35:43.728 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c09ffb640 1 -- 172.21.15.161:0/2538266631 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c10007260 con 0x7f3c1406ead0 2024-03-23T14:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.727+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c1414ff10 con 0x7f3c1406ead0 2024-03-23T14:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.728+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c140735e0 con 0x7f3c1406ead0 2024-03-23T14:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.729+0000 7f3c09ffb640 1 -- 172.21.15.161:0/2538266631 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3c100073c0 con 0x7f3c1406ead0 2024-03-23T14:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.729+0000 7f3c09ffb640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3bf806b840 0x7f3bf806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.729+0000 7f3c09ffb640 1 -- 172.21.15.161:0/2538266631 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3c10089510 con 0x7f3c1406ead0 2024-03-23T14:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.729+0000 7f3c188e6640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3bf806b840 0x7f3bf806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.730+0000 7f3c188e6640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3bf806b840 0x7f3bf806dd00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3c04036000 tx=0x7f3c0403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:43.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.733+0000 7f3c09ffb640 1 -- 172.21.15.161:0/2538266631 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3c10053fe0 con 0x7f3c1406ead0 2024-03-23T14:35:43.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.906+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c1404f550 con 0x7f3c1406ead0 2024-03-23T14:35:43.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.907+0000 7f3c09ffb640 1 -- 172.21.15.161:0/2538266631 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f3c10087e60 con 0x7f3c1406ead0 2024-03-23T14:35:43.904 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:43.904 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3bf806b840 msgr2=0x7f3bf806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3bf806b840 0x7f3bf806dd00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3c04036000 tx=0x7f3c0403b040 comp rx=0 tx=0).stop 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1406ead0 msgr2=0x7f3c1414f9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1406ead0 0x7f3c1414f9d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f3c1000c400 tx=0x7f3c1000c8c0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 shutdown_connections 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3bf806b840 0x7f3bf806dd00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3c140724e0 0x7f3c14150520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3c14072100 0x7f3c14138380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 --2- 172.21.15.161:0/2538266631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3c1406ead0 0x7f3c1414f9d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:43.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 >> 172.21.15.161:0/2538266631 conn(0x7f3c140b17f0 msgr2=0x7f3c1415c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:43.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.909+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 shutdown_connections 2024-03-23T14:35:43.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:43.910+0000 7f3c1b34b640 1 -- 172.21.15.161:0/2538266631 wait complete. 2024-03-23T14:35:43.919 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:43.920 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:44.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.027+0000 7eff8f781640 1 Processor -- start 2024-03-23T14:35:44.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.027+0000 7eff8f781640 1 -- start start 2024-03-23T14:35:44.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8f781640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7eff880d1270 0x7eff880d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8f781640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d29d0 0x7eff880b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8f781640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff88161c00 con 0x7eff88161820 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8f781640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7eff880d2e20 con 0x7eff880d1270 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8f781640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7eff880cea30 con 0x7eff880d29d0 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8cd1c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7eff880d1270 0x7eff880d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8d51d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7eff88161820 0x7eff880d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60978/0 (socket says 172.21.15.161:60978) 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8d51d640 1 -- 172.21.15.161:0/510814185 learned_addr learned my addr 172.21.15.161:0/510814185 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.028+0000 7eff8dd1e640 1 --2- 172.21.15.161:0/510814185 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d29d0 0x7eff880b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.029+0000 7eff7ffff640 1 -- 172.21.15.161:0/510814185 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3624438166 0 0) 0x7eff88161c00 con 0x7eff88161820 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.029+0000 7eff7ffff640 1 -- 172.21.15.161:0/510814185 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff74003520 con 0x7eff88161820 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.029+0000 7eff8cd1c640 1 -- 172.21.15.161:0/510814185 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d29d0 msgr2=0x7eff880b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.029+0000 7eff8cd1c640 1 --2- 172.21.15.161:0/510814185 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d29d0 0x7eff880b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.029+0000 7eff8cd1c640 1 -- 172.21.15.161:0/510814185 >> v1:172.21.15.161:6789/0 conn(0x7eff88161820 legacy=0x7eff880d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.029+0000 7eff8cd1c640 1 -- 172.21.15.161:0/510814185 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff880cfd70 con 0x7eff880d1270 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.029+0000 7eff8cd1c640 1 --2- 172.21.15.161:0/510814185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff880d1270 0x7eff880d2490 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7eff780097a0 tx=0x7eff7802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb3e48ecdf58f12 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff7ffff640 1 -- 172.21.15.161:0/510814185 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff78031830 con 0x7eff880d1270 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff7ffff640 1 -- 172.21.15.161:0/510814185 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff78031990 con 0x7eff880d1270 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff7ffff640 1 -- 172.21.15.161:0/510814185 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff78031d20 con 0x7eff880d1270 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff8f781640 1 -- 172.21.15.161:0/510814185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff880d1270 msgr2=0x7eff880d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff8f781640 1 --2- 172.21.15.161:0/510814185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff880d1270 0x7eff880d2490 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7eff780097a0 tx=0x7eff7802b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff8f781640 1 -- 172.21.15.161:0/510814185 shutdown_connections 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff8f781640 1 --2- 172.21.15.161:0/510814185 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d29d0 0x7eff880b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff8f781640 1 --2- 172.21.15.161:0/510814185 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff880d1270 0x7eff880d2490 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff8f781640 1 -- 172.21.15.161:0/510814185 >> 172.21.15.161:0/510814185 conn(0x7eff8809d190 msgr2=0x7eff880d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:44.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff8f781640 1 -- 172.21.15.161:0/510814185 shutdown_connections 2024-03-23T14:35:44.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.030+0000 7eff8f781640 1 -- 172.21.15.161:0/510814185 wait complete. 2024-03-23T14:35:44.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.031+0000 7eff8f781640 1 Processor -- start 2024-03-23T14:35:44.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.031+0000 7eff8f781640 1 -- start start 2024-03-23T14:35:44.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8f781640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d1270 0x7eff8815c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8f781640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff880d29d0 0x7eff880caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8f781640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff88161820 0x7eff880cb4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8f781640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff880cbad0 con 0x7eff88161820 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8f781640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7eff880cbdb0 con 0x7eff880d29d0 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8f781640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7eff880cc7c0 con 0x7eff880d1270 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8dd1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff88161820 0x7eff880cb4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8dd1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff88161820 0x7eff880cb4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41530/0 (socket says 172.21.15.161:41530) 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8dd1e640 1 -- 172.21.15.161:0/130334439 learned_addr learned my addr 172.21.15.161:0/130334439 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8d51d640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d1270 0x7eff8815c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8cd1c640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff880d29d0 0x7eff880caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8dd1e640 1 -- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d1270 msgr2=0x7eff8815c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8dd1e640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d1270 0x7eff8815c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8dd1e640 1 -- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff880d29d0 msgr2=0x7eff880caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8dd1e640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff880d29d0 0x7eff880caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.032+0000 7eff8dd1e640 1 -- 172.21.15.161:0/130334439 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff78009400 con 0x7eff88161820 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.033+0000 7eff8dd1e640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff88161820 0x7eff880cb4c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff8000e7f0 tx=0x7eff8000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.033+0000 7eff7dffb640 1 -- 172.21.15.161:0/130334439 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff80009510 con 0x7eff88161820 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.033+0000 7eff7dffb640 1 -- 172.21.15.161:0/130334439 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff8000ee40 con 0x7eff88161820 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.033+0000 7eff7dffb640 1 -- 172.21.15.161:0/130334439 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff8000cb80 con 0x7eff88161820 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.033+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff880ccaa0 con 0x7eff88161820 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.033+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff881f39f0 con 0x7eff88161820 2024-03-23T14:35:44.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.034+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff880ccce0 con 0x7eff88161820 2024-03-23T14:35:44.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.034+0000 7eff7dffb640 1 -- 172.21.15.161:0/130334439 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff8000cce0 con 0x7eff88161820 2024-03-23T14:35:44.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.035+0000 7eff7dffb640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6c06b810 0x7eff6c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.035+0000 7eff7dffb640 1 -- 172.21.15.161:0/130334439 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7eff8008a240 con 0x7eff88161820 2024-03-23T14:35:44.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.035+0000 7eff8d51d640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6c06b810 0x7eff6c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.032 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.035+0000 7eff8d51d640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6c06b810 0x7eff6c06dcd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7eff84005c00 tx=0x7eff84009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.035 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.039+0000 7eff7dffb640 1 -- 172.21.15.161:0/130334439 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff880ccce0 con 0x7eff88161820 2024-03-23T14:35:44.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.212+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7eff880ccce0 con 0x7eff88161820 2024-03-23T14:35:44.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.213+0000 7eff7dffb640 1 -- 172.21.15.161:0/130334439 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7eff880ccce0 con 0x7eff88161820 2024-03-23T14:35:44.209 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:44.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6c06b810 msgr2=0x7eff6c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6c06b810 0x7eff6c06dcd0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7eff84005c00 tx=0x7eff84009040 comp rx=0 tx=0).stop 2024-03-23T14:35:44.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff88161820 msgr2=0x7eff880cb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff88161820 0x7eff880cb4c0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7eff8000e7f0 tx=0x7eff8000ebf0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 shutdown_connections 2024-03-23T14:35:44.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6c06b810 0x7eff6c06dcd0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff88161820 0x7eff880cb4c0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff880d29d0 0x7eff880caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 --2- 172.21.15.161:0/130334439 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff880d1270 0x7eff8815c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 >> 172.21.15.161:0/130334439 conn(0x7eff8809d190 msgr2=0x7eff880b1310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:44.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 shutdown_connections 2024-03-23T14:35:44.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.215+0000 7eff8f781640 1 -- 172.21.15.161:0/130334439 wait complete. 2024-03-23T14:35:44.222 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:44.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.330+0000 7fe0ee193640 1 Processor -- start 2024-03-23T14:35:44.326 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.330+0000 7fe0ee193640 1 -- start start 2024-03-23T14:35:44.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.330+0000 7fe0ee193640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe0e8145520 0x7fe0e8160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.330+0000 7fe0ee193640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e80d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.330+0000 7fe0ee193640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0e80d0880 con 0x7fe0e8070c20 2024-03-23T14:35:44.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.330+0000 7fe0ee193640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fe0e81608c0 con 0x7fe0e8145520 2024-03-23T14:35:44.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.330+0000 7fe0ee193640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e80ce410 con 0x7fe0e80cfee0 2024-03-23T14:35:44.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe0e8070c20 0x7fe0e80710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60990/0 (socket says 172.21.15.161:60990) 2024-03-23T14:35:44.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e7fff640 1 -- 172.21.15.161:0/1046231118 learned_addr learned my addr 172.21.15.161:0/1046231118 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:44.327 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e77fe640 1 --2- 172.21.15.161:0/1046231118 >> v2:172.21.15.183:3300/0 conn(0x7fe0e8145520 0x7fe0e8160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e6ffd640 1 --2- 172.21.15.161:0/1046231118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e80d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e67fc640 1 -- 172.21.15.161:0/1046231118 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1476485592 0 0) 0x7fe0e80d0880 con 0x7fe0e8070c20 2024-03-23T14:35:44.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e67fc640 1 -- 172.21.15.161:0/1046231118 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe0d0003520 con 0x7fe0e8070c20 2024-03-23T14:35:44.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e6ffd640 1 -- 172.21.15.161:0/1046231118 >> v2:172.21.15.183:3300/0 conn(0x7fe0e8145520 msgr2=0x7fe0e8160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:44.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e6ffd640 1 --2- 172.21.15.161:0/1046231118 >> v2:172.21.15.183:3300/0 conn(0x7fe0e8145520 0x7fe0e8160380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e6ffd640 1 -- 172.21.15.161:0/1046231118 >> v1:172.21.15.161:6789/0 conn(0x7fe0e8070c20 legacy=0x7fe0e80710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e6ffd640 1 -- 172.21.15.161:0/1046231118 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0e80ce6f0 con 0x7fe0e80cfee0 2024-03-23T14:35:44.328 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.331+0000 7fe0e77fe640 1 --2- 172.21.15.161:0/1046231118 >> v2:172.21.15.183:3300/0 conn(0x7fe0e8145520 0x7fe0e8160380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:44.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0e6ffd640 1 --2- 172.21.15.161:0/1046231118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e80d0340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0097a0 tx=0x7fe0dc02b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4a9e514e95961b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0e67fc640 1 -- 172.21.15.161:0/1046231118 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc02e070 con 0x7fe0e80cfee0 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0e67fc640 1 -- 172.21.15.161:0/1046231118 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0dc0315a0 con 0x7fe0e80cfee0 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0e67fc640 1 -- 172.21.15.161:0/1046231118 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc031910 con 0x7fe0e80cfee0 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0ee193640 1 -- 172.21.15.161:0/1046231118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 msgr2=0x7fe0e80d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0ee193640 1 --2- 172.21.15.161:0/1046231118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e80d0340 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0097a0 tx=0x7fe0dc02b840 comp rx=0 tx=0).stop 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0ee193640 1 -- 172.21.15.161:0/1046231118 shutdown_connections 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0ee193640 1 --2- 172.21.15.161:0/1046231118 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e80d0340 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0ee193640 1 --2- 172.21.15.161:0/1046231118 >> v2:172.21.15.183:3300/0 conn(0x7fe0e8145520 0x7fe0e8160380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0ee193640 1 -- 172.21.15.161:0/1046231118 >> 172.21.15.161:0/1046231118 conn(0x7fe0e806b4a0 msgr2=0x7fe0e806b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.332+0000 7fe0ee193640 1 -- 172.21.15.161:0/1046231118 shutdown_connections 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.333+0000 7fe0ee193640 1 -- 172.21.15.161:0/1046231118 wait complete. 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.334+0000 7fe0ee193640 1 Processor -- start 2024-03-23T14:35:44.330 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.334+0000 7fe0ee193640 1 -- start start 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.334+0000 7fe0ee193640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0e8070c20 0x7fe0e814f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.334+0000 7fe0ee193640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e806d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0ee193640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e8145520 0x7fe0e806db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0ee193640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e80af220 con 0x7fe0e8145520 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0ee193640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe0e80ae6a0 con 0x7fe0e8070c20 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0ee193640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe0e80ae980 con 0x7fe0e80cfee0 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e8145520 0x7fe0e806db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e8145520 0x7fe0e806db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41544/0 (socket says 172.21.15.161:41544) 2024-03-23T14:35:44.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e77fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0e8070c20 0x7fe0e814f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 -- 172.21.15.161:0/2687048600 learned_addr learned my addr 172.21.15.161:0/2687048600 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e6ffd640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e806d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 -- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 msgr2=0x7fe0e806d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e806d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 -- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0e8070c20 msgr2=0x7fe0e814f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0e8070c20 0x7fe0e814f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 -- 172.21.15.161:0/2687048600 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0d8009400 con 0x7fe0e8145520 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e77fe640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0e8070c20 0x7fe0e814f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0e7fff640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e8145520 0x7fe0e806db50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe0e000cb40 tx=0x7fe0e000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0cffff640 1 -- 172.21.15.161:0/2687048600 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e000aca0 con 0x7fe0e8145520 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0cffff640 1 -- 172.21.15.161:0/2687048600 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0e000ae00 con 0x7fe0e8145520 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0dc009400 con 0x7fe0e8145520 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0cffff640 1 -- 172.21.15.161:0/2687048600 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0e0012620 con 0x7fe0e8145520 2024-03-23T14:35:44.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.335+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0e80aef40 con 0x7fe0e8145520 2024-03-23T14:35:44.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.336+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0e8157f30 con 0x7fe0e8145520 2024-03-23T14:35:44.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.337+0000 7fe0cffff640 1 -- 172.21.15.161:0/2687048600 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe0e0007070 con 0x7fe0e8145520 2024-03-23T14:35:44.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.337+0000 7fe0cffff640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0d406b810 0x7fe0d406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.337+0000 7fe0cffff640 1 -- 172.21.15.161:0/2687048600 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fe0e008a1d0 con 0x7fe0e8145520 2024-03-23T14:35:44.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.337+0000 7fe0e77fe640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0d406b810 0x7fe0d406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.338+0000 7fe0e77fe640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0d406b810 0x7fe0d406dcd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8009fd0 tx=0x7fe0d8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.341+0000 7fe0cffff640 1 -- 172.21.15.161:0/2687048600 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe0e8157f30 con 0x7fe0e8145520 2024-03-23T14:35:44.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.513+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe0e8157f30 con 0x7fe0e8145520 2024-03-23T14:35:44.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.513+0000 7fe0cffff640 1 -- 172.21.15.161:0/2687048600 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7fe0e8157f30 con 0x7fe0e8145520 2024-03-23T14:35:44.509 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:44.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0d406b810 msgr2=0x7fe0d406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0d406b810 0x7fe0d406dcd0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe0d8009fd0 tx=0x7fe0d8009190 comp rx=0 tx=0).stop 2024-03-23T14:35:44.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e8145520 msgr2=0x7fe0e806db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e8145520 0x7fe0e806db50 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fe0e000cb40 tx=0x7fe0e000a060 comp rx=0 tx=0).stop 2024-03-23T14:35:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 shutdown_connections 2024-03-23T14:35:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe0d406b810 0x7fe0d406dcd0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe0e8145520 0x7fe0e806db50 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe0e80cfee0 0x7fe0e806d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 --2- 172.21.15.161:0/2687048600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe0e8070c20 0x7fe0e814f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 >> 172.21.15.161:0/2687048600 conn(0x7fe0e806b4a0 msgr2=0x7fe0e8142b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 shutdown_connections 2024-03-23T14:35:44.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.515+0000 7fe0ee193640 1 -- 172.21.15.161:0/2687048600 wait complete. 2024-03-23T14:35:44.522 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:44.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.629+0000 7f1810197640 1 Processor -- start 2024-03-23T14:35:44.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.630+0000 7f1810197640 1 -- start start 2024-03-23T14:35:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.630+0000 7f1810197640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18080b2aa0 0x7f18080cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.630+0000 7f1810197640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18080d0600 con 0x7f18080d01b0 2024-03-23T14:35:44.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.630+0000 7f1810197640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f180815f480 con 0x7f18081552f0 2024-03-23T14:35:44.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.630+0000 7f1810197640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f18081556d0 con 0x7f18080b2aa0 2024-03-23T14:35:44.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180d732640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18080b2aa0 0x7f18080cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180e734640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f18080d01b0 0x7f180806ded0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32774/0 (socket says 172.21.15.161:32774) 2024-03-23T14:35:44.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180df33640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f18081552f0 0x7f18080b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:44190/0 (socket says 172.21.15.161:44190) 2024-03-23T14:35:44.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180d732640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18080b2aa0 0x7f18080cfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41552/0 (socket says 172.21.15.161:41552) 2024-03-23T14:35:44.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180e734640 1 -- 172.21.15.161:0/971422318 learned_addr learned my addr 172.21.15.161:0/971422318 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:44.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180d732640 1 -- 172.21.15.161:0/971422318 >> v1:172.21.15.183:6789/0 conn(0x7f18081552f0 legacy=0x7f18080b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180d732640 1 -- 172.21.15.161:0/971422318 >> v1:172.21.15.161:6789/0 conn(0x7f18080d01b0 legacy=0x7f180806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180cf31640 1 -- 172.21.15.161:0/971422318 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1644540469 0 0) 0x7f180815f480 con 0x7f18081552f0 2024-03-23T14:35:44.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.631+0000 7f180d732640 1 -- 172.21.15.161:0/971422318 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180815e870 con 0x7f18080b2aa0 2024-03-23T14:35:44.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f180d732640 1 --2- 172.21.15.161:0/971422318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18080b2aa0 0x7f18080cfc70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0097a0 tx=0x7f17fc02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb6e789655a7402a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f180cf31640 1 -- 172.21.15.161:0/971422318 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc030710 con 0x7f18080b2aa0 2024-03-23T14:35:44.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f180cf31640 1 -- 172.21.15.161:0/971422318 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17fc02e050 con 0x7f18080b2aa0 2024-03-23T14:35:44.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f180cf31640 1 -- 172.21.15.161:0/971422318 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17fc030a90 con 0x7f18080b2aa0 2024-03-23T14:35:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f1810197640 1 -- 172.21.15.161:0/971422318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18080b2aa0 msgr2=0x7f18080cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f1810197640 1 --2- 172.21.15.161:0/971422318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18080b2aa0 0x7f18080cfc70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f17fc0097a0 tx=0x7f17fc02b790 comp rx=0 tx=0).stop 2024-03-23T14:35:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f1810197640 1 -- 172.21.15.161:0/971422318 shutdown_connections 2024-03-23T14:35:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f1810197640 1 --2- 172.21.15.161:0/971422318 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18080b2aa0 0x7f18080cfc70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f1810197640 1 -- 172.21.15.161:0/971422318 >> 172.21.15.161:0/971422318 conn(0x7f180806b260 msgr2=0x7f180806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f1810197640 1 -- 172.21.15.161:0/971422318 shutdown_connections 2024-03-23T14:35:44.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.632+0000 7f1810197640 1 -- 172.21.15.161:0/971422318 wait complete. 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.633+0000 7f1810197640 1 Processor -- start 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f1810197640 1 -- start start 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f1810197640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18080b2aa0 0x7f18080b0570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f1810197640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18080d01b0 0x7f180813c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f1810197640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18081552f0 0x7f18080ce3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f1810197640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1808158760 con 0x7f18081552f0 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f1810197640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f180813c960 con 0x7f18080d01b0 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f1810197640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f18080af7b0 con 0x7f18080b2aa0 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f180e734640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18081552f0 0x7f18080ce3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f180d732640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18080d01b0 0x7f180813c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f180e734640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18081552f0 0x7f18080ce3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41558/0 (socket says 172.21.15.161:41558) 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f180e734640 1 -- 172.21.15.161:0/2198556835 learned_addr learned my addr 172.21.15.161:0/2198556835 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.634+0000 7f180df33640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18080b2aa0 0x7f18080b0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f180e734640 1 -- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18080b2aa0 msgr2=0x7f18080b0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f180e734640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18080b2aa0 0x7f18080b0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f180e734640 1 -- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18080d01b0 msgr2=0x7f180813c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f180e734640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18080d01b0 0x7f180813c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f180e734640 1 -- 172.21.15.161:0/2198556835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17fc009400 con 0x7f18081552f0 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f180e734640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18081552f0 0x7f18080ce3b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f17f800c750 tx=0x7f17f800cc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f18067fc640 1 -- 172.21.15.161:0/2198556835 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f80074e0 con 0x7f18081552f0 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f18067fc640 1 -- 172.21.15.161:0/2198556835 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f17f8007640 con 0x7f18081552f0 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f18067fc640 1 -- 172.21.15.161:0/2198556835 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17f8004a00 con 0x7f18081552f0 2024-03-23T14:35:44.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18080afa90 con 0x7f18081552f0 2024-03-23T14:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.635+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18080b0050 con 0x7f18081552f0 2024-03-23T14:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.636+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18080cf8a0 con 0x7f18081552f0 2024-03-23T14:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.636+0000 7f18067fc640 1 -- 172.21.15.161:0/2198556835 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f17f8004b60 con 0x7f18081552f0 2024-03-23T14:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.637+0000 7f18067fc640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17f406b810 0x7f17f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.637+0000 7f18067fc640 1 -- 172.21.15.161:0/2198556835 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f17f80896d0 con 0x7f18081552f0 2024-03-23T14:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.637+0000 7f180df33640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17f406b810 0x7f17f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.637+0000 7f180df33640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17f406b810 0x7f17f406dcd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18000012e0 tx=0x7f1800001630 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.641+0000 7f18067fc640 1 -- 172.21.15.161:0/2198556835 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f17f8054230 con 0x7f18081552f0 2024-03-23T14:35:44.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.814+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f18080d0600 con 0x7f18081552f0 2024-03-23T14:35:44.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.814+0000 7f18067fc640 1 -- 172.21.15.161:0/2198556835 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f18080d0600 con 0x7f18081552f0 2024-03-23T14:35:44.810 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:44.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17f406b810 msgr2=0x7f17f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17f406b810 0x7f17f406dcd0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f18000012e0 tx=0x7f1800001630 comp rx=0 tx=0).stop 2024-03-23T14:35:44.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18081552f0 msgr2=0x7f18080ce3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18081552f0 0x7f18080ce3b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f17f800c750 tx=0x7f17f800cc10 comp rx=0 tx=0).stop 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 shutdown_connections 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17f406b810 0x7f17f406dcd0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18081552f0 0x7f18080ce3b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18080d01b0 0x7f180813c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 --2- 172.21.15.161:0/2198556835 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18080b2aa0 0x7f18080b0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 >> 172.21.15.161:0/2198556835 conn(0x7f180806b260 msgr2=0x7f1808159790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 shutdown_connections 2024-03-23T14:35:44.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.816+0000 7f1810197640 1 -- 172.21.15.161:0/2198556835 wait complete. 2024-03-23T14:35:44.823 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:35:44.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.930+0000 7f40c16c9640 1 Processor -- start 2024-03-23T14:35:44.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.930+0000 7f40c16c9640 1 -- start start 2024-03-23T14:35:44.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40c16c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0ad850 0x7f40bc0cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40c16c9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f40bc0cd380 0x7f40bc168810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40c16c9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40bc02c1c0 con 0x7f40bc0cf840 2024-03-23T14:35:44.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40c16c9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f40bc02c650 con 0x7f40bc0cd380 2024-03-23T14:35:44.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40c16c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f40bc15e620 con 0x7f40bc0ad850 2024-03-23T14:35:44.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40baffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f40bc0cf840 0x7f40bc0ad140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32788/0 (socket says 172.21.15.161:32788) 2024-03-23T14:35:44.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40baffd640 1 -- 172.21.15.161:0/226707819 learned_addr learned my addr 172.21.15.161:0/226707819 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:44.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40b27fc640 1 --2- 172.21.15.161:0/226707819 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0ad850 0x7f40bc0cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40bb7fe640 1 --2- 172.21.15.161:0/226707819 >> v2:172.21.15.183:3300/0 conn(0x7f40bc0cd380 0x7f40bc168810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40ba7fc640 1 -- 172.21.15.161:0/226707819 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1954641495 0 0) 0x7f40bc02c1c0 con 0x7f40bc0cf840 2024-03-23T14:35:44.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.931+0000 7f40ba7fc640 1 -- 172.21.15.161:0/226707819 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f40a8003520 con 0x7f40bc0cf840 2024-03-23T14:35:44.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.932+0000 7f40bb7fe640 1 -- 172.21.15.161:0/226707819 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0ad850 msgr2=0x7f40bc0cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:44.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.932+0000 7f40bb7fe640 1 --2- 172.21.15.161:0/226707819 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0ad850 0x7f40bc0cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.932+0000 7f40bb7fe640 1 -- 172.21.15.161:0/226707819 >> v1:172.21.15.161:6789/0 conn(0x7f40bc0cf840 legacy=0x7f40bc0ad140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.932+0000 7f40bb7fe640 1 -- 172.21.15.161:0/226707819 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40bc071630 con 0x7f40bc0cd380 2024-03-23T14:35:44.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.932+0000 7f40bb7fe640 1 --2- 172.21.15.161:0/226707819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cd380 0x7f40bc168810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009d90 tx=0x7f40b402bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6faef3e44f62285b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.932+0000 7f40ba7fc640 1 -- 172.21.15.161:0/226707819 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b402cc50 con 0x7f40bc0cd380 2024-03-23T14:35:44.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.932+0000 7f40ba7fc640 1 -- 172.21.15.161:0/226707819 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40b402cdb0 con 0x7f40bc0cd380 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.932+0000 7f40ba7fc640 1 -- 172.21.15.161:0/226707819 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40b4034660 con 0x7f40bc0cd380 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.933+0000 7f40c16c9640 1 -- 172.21.15.161:0/226707819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cd380 msgr2=0x7f40bc168810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.933+0000 7f40c16c9640 1 --2- 172.21.15.161:0/226707819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cd380 0x7f40bc168810 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f40b4009d90 tx=0x7f40b402bea0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.933+0000 7f40c16c9640 1 -- 172.21.15.161:0/226707819 shutdown_connections 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.933+0000 7f40c16c9640 1 --2- 172.21.15.161:0/226707819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cd380 0x7f40bc168810 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.933+0000 7f40c16c9640 1 --2- 172.21.15.161:0/226707819 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0ad850 0x7f40bc0cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.933+0000 7f40c16c9640 1 -- 172.21.15.161:0/226707819 >> 172.21.15.161:0/226707819 conn(0x7f40bc0b1e60 msgr2=0x7f40bc0b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.933+0000 7f40c16c9640 1 -- 172.21.15.161:0/226707819 shutdown_connections 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.933+0000 7f40c16c9640 1 -- 172.21.15.161:0/226707819 wait complete. 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.934+0000 7f40c16c9640 1 Processor -- start 2024-03-23T14:35:44.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.934+0000 7f40c16c9640 1 -- start start 2024-03-23T14:35:44.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40c16c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40bc0ad850 0x7f40bc158830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40c16c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0cd380 0x7f40bc0c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40c16c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cf840 0x7f40bc0b3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40c16c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc06dab0 con 0x7f40bc0ad850 2024-03-23T14:35:44.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40c16c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f40bc0c5630 con 0x7f40bc0cf840 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40c16c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f40bc0c5910 con 0x7f40bc0cd380 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40baffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40bc0ad850 0x7f40bc158830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40baffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40bc0ad850 0x7f40bc158830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41560/0 (socket says 172.21.15.161:41560) 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40baffd640 1 -- 172.21.15.161:0/1711027113 learned_addr learned my addr 172.21.15.161:0/1711027113 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40bb7fe640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cf840 0x7f40bc0b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.935+0000 7f40b27fc640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0cd380 0x7f40bc0c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40baffd640 1 -- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0cd380 msgr2=0x7f40bc0c66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40baffd640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0cd380 0x7f40bc0c66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40baffd640 1 -- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cf840 msgr2=0x7f40bc0b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40baffd640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cf840 0x7f40bc0b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40baffd640 1 -- 172.21.15.161:0/1711027113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40b4009a70 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40baffd640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40bc0ad850 0x7f40bc158830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00c390 tx=0x7f40ac00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40b3fff640 1 -- 172.21.15.161:0/1711027113 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ac005ce0 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40b3fff640 1 -- 172.21.15.161:0/1711027113 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40ac005e40 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40bc0c5bf0 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40b3fff640 1 -- 172.21.15.161:0/1711027113 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40ac0072f0 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.936+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40bc0c61b0 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.937+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40bc15d7a0 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.937+0000 7f40b3fff640 1 -- 172.21.15.161:0/1711027113 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f40ac007450 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.938+0000 7f40b3fff640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f409c06b810 0x7f409c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.938+0000 7f40b3fff640 1 -- 172.21.15.161:0/1711027113 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f40ac0896e0 con 0x7f40bc0ad850 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.938+0000 7f40b27fc640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f409c06b810 0x7f409c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:44.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.938+0000 7f40b27fc640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f409c06b810 0x7f409c06dcd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40bc158d70 tx=0x7f40a4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:44.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:44.942+0000 7f40b3fff640 1 -- 172.21.15.161:0/1711027113 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f40ac054240 con 0x7f40bc0ad850 2024-03-23T14:35:45.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.124+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f40bc0c70d0 con 0x7f40bc0ad850 2024-03-23T14:35:45.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.125+0000 7f40b3fff640 1 -- 172.21.15.161:0/1711027113 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f40ac057900 con 0x7f40bc0ad850 2024-03-23T14:35:45.121 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:35:45.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f409c06b810 msgr2=0x7f409c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f409c06b810 0x7f409c06dcd0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f40bc158d70 tx=0x7f40a4009190 comp rx=0 tx=0).stop 2024-03-23T14:35:45.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40bc0ad850 msgr2=0x7f40bc158830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40bc0ad850 0x7f40bc158830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f40ac00c390 tx=0x7f40ac00c850 comp rx=0 tx=0).stop 2024-03-23T14:35:45.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 shutdown_connections 2024-03-23T14:35:45.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f409c06b810 0x7f409c06dcd0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40bc0cf840 0x7f40bc0b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40bc0cd380 0x7f40bc0c66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 --2- 172.21.15.161:0/1711027113 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40bc0ad850 0x7f40bc158830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 >> 172.21.15.161:0/1711027113 conn(0x7f40bc0b1e60 msgr2=0x7f40bc06e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:45.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 shutdown_connections 2024-03-23T14:35:45.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.127+0000 7f40c16c9640 1 -- 172.21.15.161:0/1711027113 wait complete. 2024-03-23T14:35:45.134 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:35:45.134 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:45.150 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.256+0000 7fa54a407640 1 Processor -- start 2024-03-23T14:35:45.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.256+0000 7fa54a407640 1 -- start start 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa54a407640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa544072100 0x7fa54415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa54a407640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5440724e0 0x7fa54406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa54a407640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa54415be70 con 0x7fa54415b930 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa54a407640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa54415c430 con 0x7fa544072100 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa54a407640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa54415c150 con 0x7fa5440724e0 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa543fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa544072100 0x7fa54415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa543fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa544072100 0x7fa54415b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41568/0 (socket says 172.21.15.161:41568) 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa543fff640 1 -- 172.21.15.161:0/1766840239 learned_addr learned my addr 172.21.15.161:0/1766840239 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa5437fe640 1 --2- 172.21.15.161:0/1766840239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5440724e0 0x7fa54406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa543fff640 1 -- 172.21.15.161:0/1766840239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5440724e0 msgr2=0x7fa54406e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:45.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa543fff640 1 --2- 172.21.15.161:0/1766840239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5440724e0 0x7fa54406e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa543fff640 1 -- 172.21.15.161:0/1766840239 >> v1:172.21.15.183:6789/0 conn(0x7fa54415b930 legacy=0x7fa54415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.257+0000 7fa543fff640 1 -- 172.21.15.161:0/1766840239 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5440b2c70 con 0x7fa544072100 2024-03-23T14:35:45.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.258+0000 7fa543fff640 1 --2- 172.21.15.161:0/1766840239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa54415b3f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa53400bb10 tx=0x7fa53402e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95f885a586632f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.258+0000 7fa542ffd640 1 -- 172.21.15.161:0/1766840239 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534036440 con 0x7fa544072100 2024-03-23T14:35:45.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.258+0000 7fa542ffd640 1 -- 172.21.15.161:0/1766840239 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5340365a0 con 0x7fa544072100 2024-03-23T14:35:45.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.258+0000 7fa542ffd640 1 -- 172.21.15.161:0/1766840239 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534036930 con 0x7fa544072100 2024-03-23T14:35:45.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.259+0000 7fa54a407640 1 -- 172.21.15.161:0/1766840239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 msgr2=0x7fa54415b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.259+0000 7fa54a407640 1 --2- 172.21.15.161:0/1766840239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa54415b3f0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fa53400bb10 tx=0x7fa53402e150 comp rx=0 tx=0).stop 2024-03-23T14:35:45.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.259+0000 7fa54a407640 1 -- 172.21.15.161:0/1766840239 shutdown_connections 2024-03-23T14:35:45.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.259+0000 7fa54a407640 1 --2- 172.21.15.161:0/1766840239 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5440724e0 0x7fa54406e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.259+0000 7fa54a407640 1 --2- 172.21.15.161:0/1766840239 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa54415b3f0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.259+0000 7fa54a407640 1 -- 172.21.15.161:0/1766840239 >> 172.21.15.161:0/1766840239 conn(0x7fa5440b17f0 msgr2=0x7fa5440b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.259+0000 7fa54a407640 1 -- 172.21.15.161:0/1766840239 shutdown_connections 2024-03-23T14:35:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.259+0000 7fa54a407640 1 -- 172.21.15.161:0/1766840239 wait complete. 2024-03-23T14:35:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.260+0000 7fa54a407640 1 Processor -- start 2024-03-23T14:35:45.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.260+0000 7fa54a407640 1 -- start start 2024-03-23T14:35:45.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa54a407640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa5441471b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa54a407640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5440724e0 0x7fa5441e56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa54a407640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa54415b930 0x7fa5441e5c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa54a407640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa544143270 con 0x7fa544072100 2024-03-23T14:35:45.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa54a407640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa544145420 con 0x7fa5440724e0 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa54a407640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa544141130 con 0x7fa54415b930 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa543fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa5441471b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa543fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa5441471b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41574/0 (socket says 172.21.15.161:41574) 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa543fff640 1 -- 172.21.15.161:0/3222710008 learned_addr learned my addr 172.21.15.161:0/3222710008 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa5489a4640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa54415b930 0x7fa5441e5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa5437fe640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5440724e0 0x7fa5441e56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa543fff640 1 -- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa54415b930 msgr2=0x7fa5441e5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa543fff640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa54415b930 0x7fa5441e5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa543fff640 1 -- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5440724e0 msgr2=0x7fa5441e56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa543fff640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5440724e0 0x7fa5441e56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.261+0000 7fa543fff640 1 -- 172.21.15.161:0/3222710008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa53400b770 con 0x7fa544072100 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.262+0000 7fa543fff640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa5441471b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa53400b2c0 tx=0x7fa5340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.262+0000 7fa540ff9640 1 -- 172.21.15.161:0/3222710008 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534036440 con 0x7fa544072100 2024-03-23T14:35:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.262+0000 7fa540ff9640 1 -- 172.21.15.161:0/3222710008 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa534034b80 con 0x7fa544072100 2024-03-23T14:35:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.262+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa544144a80 con 0x7fa544072100 2024-03-23T14:35:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.262+0000 7fa540ff9640 1 -- 172.21.15.161:0/3222710008 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa534041440 con 0x7fa544072100 2024-03-23T14:35:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.262+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa544145040 con 0x7fa544072100 2024-03-23T14:35:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.263+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5440735e0 con 0x7fa544072100 2024-03-23T14:35:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.263+0000 7fa540ff9640 1 -- 172.21.15.161:0/3222710008 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa53403d040 con 0x7fa544072100 2024-03-23T14:35:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.264+0000 7fa540ff9640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa53006b810 0x7fa53006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.264+0000 7fa540ff9640 1 -- 172.21.15.161:0/3222710008 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa5340aede0 con 0x7fa544072100 2024-03-23T14:35:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.264+0000 7fa5437fe640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa53006b810 0x7fa53006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.264+0000 7fa5437fe640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa53006b810 0x7fa53006dcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa54406ef40 tx=0x7fa538009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.268+0000 7fa540ff9640 1 -- 172.21.15.161:0/3222710008 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa534079940 con 0x7fa544072100 2024-03-23T14:35:45.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.439+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa54404f550 con 0x7fa544072100 2024-03-23T14:35:45.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.440+0000 7fa540ff9640 1 -- 172.21.15.161:0/3222710008 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7fa53407d000 con 0x7fa544072100 2024-03-23T14:35:45.436 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:45.437 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:45.438 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa53006b810 msgr2=0x7fa53006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa53006b810 0x7fa53006dcd0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa54406ef40 tx=0x7fa538009190 comp rx=0 tx=0).stop 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 msgr2=0x7fa5441471b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa5441471b0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fa53400b2c0 tx=0x7fa5340046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 shutdown_connections 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa53006b810 0x7fa53006dcd0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa54415b930 0x7fa5441e5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5440724e0 0x7fa5441e56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 --2- 172.21.15.161:0/3222710008 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa544072100 0x7fa5441471b0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.442+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 >> 172.21.15.161:0/3222710008 conn(0x7fa5440b17f0 msgr2=0x7fa54415dc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.443+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 shutdown_connections 2024-03-23T14:35:45.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.443+0000 7fa54a407640 1 -- 172.21.15.161:0/3222710008 wait complete. 2024-03-23T14:35:45.450 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:45.450 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:45.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.558+0000 7fc76913e640 1 Processor -- start 2024-03-23T14:35:45.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.558+0000 7fc76913e640 1 -- start start 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.559+0000 7fc76913e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc764161820 0x7fc7640d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.559+0000 7fc76913e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc764161c00 con 0x7fc7640d1270 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.559+0000 7fc76913e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7640cea30 con 0x7fc7640d2ba0 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.559+0000 7fc76913e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc7640cfd70 con 0x7fc764161820 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.559+0000 7fc762d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc764161820 0x7fc7640d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.559+0000 7fc76259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc7640d1270 0x7fc7640d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32794/0 (socket says 172.21.15.161:32794) 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.559+0000 7fc76259c640 1 -- 172.21.15.161:0/2959626335 learned_addr learned my addr 172.21.15.161:0/2959626335 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.559+0000 7fc762d9d640 1 --2- 172.21.15.161:0/2959626335 >> v2:172.21.15.161:3300/0 conn(0x7fc764161820 0x7fc7640d47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41576/0 (socket says 172.21.15.161:41576) 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.560+0000 7fc762d9d640 1 -- 172.21.15.161:0/2959626335 >> v1:172.21.15.183:6789/0 conn(0x7fc7640d2ba0 legacy=0x7fc7640b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.560+0000 7fc762d9d640 1 -- 172.21.15.161:0/2959626335 >> v1:172.21.15.161:6789/0 conn(0x7fc7640d1270 legacy=0x7fc7640d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.560+0000 7fc762d9d640 1 -- 172.21.15.161:0/2959626335 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7640cecd0 con 0x7fc764161820 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.560+0000 7fc761d9b640 1 -- 172.21.15.161:0/2959626335 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 161450233 0 0) 0x7fc764161c00 con 0x7fc7640d1270 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.560+0000 7fc762d9d640 1 --2- 172.21.15.161:0/2959626335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc764161820 0x7fc7640d47e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc754009d90 tx=0x7fc75402bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54083a9b005f9e99 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.560+0000 7fc761d9b640 1 -- 172.21.15.161:0/2959626335 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc75402cd00 con 0x7fc764161820 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.560+0000 7fc761d9b640 1 -- 172.21.15.161:0/2959626335 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc75402ce60 con 0x7fc764161820 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.560+0000 7fc761d9b640 1 -- 172.21.15.161:0/2959626335 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc754034660 con 0x7fc764161820 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.561+0000 7fc76913e640 1 -- 172.21.15.161:0/2959626335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc764161820 msgr2=0x7fc7640d47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.561+0000 7fc76913e640 1 --2- 172.21.15.161:0/2959626335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc764161820 0x7fc7640d47e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fc754009d90 tx=0x7fc75402bf50 comp rx=0 tx=0).stop 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.561+0000 7fc76913e640 1 -- 172.21.15.161:0/2959626335 shutdown_connections 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.561+0000 7fc76913e640 1 --2- 172.21.15.161:0/2959626335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc764161820 0x7fc7640d47e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.561+0000 7fc76913e640 1 -- 172.21.15.161:0/2959626335 >> 172.21.15.161:0/2959626335 conn(0x7fc76409d190 msgr2=0x7fc7640d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.561+0000 7fc76913e640 1 -- 172.21.15.161:0/2959626335 shutdown_connections 2024-03-23T14:35:45.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.561+0000 7fc76913e640 1 -- 172.21.15.161:0/2959626335 wait complete. 2024-03-23T14:35:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.562+0000 7fc76913e640 1 Processor -- start 2024-03-23T14:35:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76913e640 1 -- start start 2024-03-23T14:35:45.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76913e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7640d1270 0x7fc7640cc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76913e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7640d2ba0 0x7fc7640caf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76913e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc764161820 0x7fc76414f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76913e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc7640cce20 con 0x7fc7640d1270 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76913e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc7640cb840 con 0x7fc7640d2ba0 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76913e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc7640cbb20 con 0x7fc764161820 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc762d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7640d1270 0x7fc7640cc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc762d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7640d1270 0x7fc7640cc8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41586/0 (socket says 172.21.15.161:41586) 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc762d9d640 1 -- 172.21.15.161:0/215823220 learned_addr learned my addr 172.21.15.161:0/215823220 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76259c640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7640d2ba0 0x7fc7640caf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.563+0000 7fc76359e640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc764161820 0x7fc76414f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc762d9d640 1 -- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc764161820 msgr2=0x7fc76414f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc762d9d640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc764161820 0x7fc76414f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc762d9d640 1 -- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7640d2ba0 msgr2=0x7fc7640caf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc762d9d640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7640d2ba0 0x7fc7640caf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc762d9d640 1 -- 172.21.15.161:0/215823220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc754009a70 con 0x7fc7640d1270 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc76359e640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc764161820 0x7fc76414f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc76259c640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7640d2ba0 0x7fc7640caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc762d9d640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7640d1270 0x7fc7640cc8e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc754037040 tx=0x7fc754004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc74b7fe640 1 -- 172.21.15.161:0/215823220 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc754003b90 con 0x7fc7640d1270 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc74b7fe640 1 -- 172.21.15.161:0/215823220 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc754003d10 con 0x7fc7640d1270 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7640cbe00 con 0x7fc7640d1270 2024-03-23T14:35:45.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc74b7fe640 1 -- 172.21.15.161:0/215823220 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc754004270 con 0x7fc7640d1270 2024-03-23T14:35:45.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.564+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7640cc3c0 con 0x7fc7640d1270 2024-03-23T14:35:45.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.565+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7640d3400 con 0x7fc7640d1270 2024-03-23T14:35:45.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.566+0000 7fc74b7fe640 1 -- 172.21.15.161:0/215823220 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc75403b040 con 0x7fc7640d1270 2024-03-23T14:35:45.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.566+0000 7fc74b7fe640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc75006b810 0x7fc75006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.566+0000 7fc74b7fe640 1 -- 172.21.15.161:0/215823220 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc7540ac850 con 0x7fc7640d1270 2024-03-23T14:35:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.566+0000 7fc76259c640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc75006b810 0x7fc75006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.567+0000 7fc76259c640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc75006b810 0x7fc75006dcd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc7641611f0 tx=0x7fc758007e40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.571+0000 7fc74b7fe640 1 -- 172.21.15.161:0/215823220 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc754077320 con 0x7fc7640d1270 2024-03-23T14:35:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.742+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc76404f4e0 con 0x7fc7640d1270 2024-03-23T14:35:45.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.742+0000 7fc74b7fe640 1 -- 172.21.15.161:0/215823220 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fc754039020 con 0x7fc7640d1270 2024-03-23T14:35:45.738 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc75006b810 msgr2=0x7fc75006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc75006b810 0x7fc75006dcd0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc7641611f0 tx=0x7fc758007e40 comp rx=0 tx=0).stop 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7640d1270 msgr2=0x7fc7640cc8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7640d1270 0x7fc7640cc8e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fc754037040 tx=0x7fc754004a40 comp rx=0 tx=0).stop 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 shutdown_connections 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc75006b810 0x7fc75006dcd0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc764161820 0x7fc76414f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7640d2ba0 0x7fc7640caf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 --2- 172.21.15.161:0/215823220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7640d1270 0x7fc7640cc8e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 >> 172.21.15.161:0/215823220 conn(0x7fc76409d190 msgr2=0x7fc7640cff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:45.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 shutdown_connections 2024-03-23T14:35:45.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.744+0000 7fc76913e640 1 -- 172.21.15.161:0/215823220 wait complete. 2024-03-23T14:35:45.751 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:45.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.861+0000 7f0f376c0640 1 Processor -- start 2024-03-23T14:35:45.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.862+0000 7f0f376c0640 1 -- start start 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.862+0000 7f0f376c0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0f30161750 0x7f0f300d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.862+0000 7f0f376c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f30161b30 0x7f0f300d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.862+0000 7f0f376c0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f300d1330 con 0x7f0f300d1cb0 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.862+0000 7f0f376c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f0f300d0720 con 0x7f0f30161b30 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.862+0000 7f0f376c0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f0f300d0a00 con 0x7f0f30161750 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f35c5d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f30161b30 0x7f0f300d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f35c5d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f0f30161b30 0x7f0f300d2b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41588/0 (socket says 172.21.15.161:41588) 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f35c5d640 1 -- 172.21.15.161:0/2397131556 learned_addr learned my addr 172.21.15.161:0/2397131556 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f3545c640 1 --2- 172.21.15.161:0/2397131556 >> v2:172.21.15.183:3300/0 conn(0x7f0f30161750 0x7f0f300d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f27fff640 1 -- 172.21.15.161:0/2397131556 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4121731357 0 0) 0x7f0f300d1330 con 0x7f0f300d1cb0 2024-03-23T14:35:45.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f35c5d640 1 -- 172.21.15.161:0/2397131556 >> v2:172.21.15.183:3300/0 conn(0x7f0f30161750 msgr2=0x7f0f300d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:45.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f35c5d640 1 --2- 172.21.15.161:0/2397131556 >> v2:172.21.15.183:3300/0 conn(0x7f0f30161750 0x7f0f300d25c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f35c5d640 1 -- 172.21.15.161:0/2397131556 >> v1:172.21.15.183:6789/0 conn(0x7f0f300d1cb0 legacy=0x7f0f30162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.863+0000 7f0f35c5d640 1 -- 172.21.15.161:0/2397131556 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f300d0ce0 con 0x7f0f30161b30 2024-03-23T14:35:45.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f35c5d640 1 --2- 172.21.15.161:0/2397131556 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f30161b30 0x7f0f300d2b00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0f3008e830 tx=0x7f0f2802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c7d39f8f0dddde9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f27fff640 1 -- 172.21.15.161:0/2397131556 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2802cb20 con 0x7f0f30161b30 2024-03-23T14:35:45.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f27fff640 1 -- 172.21.15.161:0/2397131556 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f2802cc80 con 0x7f0f30161b30 2024-03-23T14:35:45.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f27fff640 1 -- 172.21.15.161:0/2397131556 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f280343f0 con 0x7f0f30161b30 2024-03-23T14:35:45.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f376c0640 1 -- 172.21.15.161:0/2397131556 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f30161b30 msgr2=0x7f0f300d2b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f376c0640 1 --2- 172.21.15.161:0/2397131556 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f30161b30 0x7f0f300d2b00 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f0f3008e830 tx=0x7f0f2802bf80 comp rx=0 tx=0).stop 2024-03-23T14:35:45.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f376c0640 1 -- 172.21.15.161:0/2397131556 shutdown_connections 2024-03-23T14:35:45.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f376c0640 1 --2- 172.21.15.161:0/2397131556 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f30161b30 0x7f0f300d2b00 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f376c0640 1 --2- 172.21.15.161:0/2397131556 >> v2:172.21.15.183:3300/0 conn(0x7f0f30161750 0x7f0f300d25c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f376c0640 1 -- 172.21.15.161:0/2397131556 >> 172.21.15.161:0/2397131556 conn(0x7f0f3009d450 msgr2=0x7f0f300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:45.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f376c0640 1 -- 172.21.15.161:0/2397131556 shutdown_connections 2024-03-23T14:35:45.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.864+0000 7f0f376c0640 1 -- 172.21.15.161:0/2397131556 wait complete. 2024-03-23T14:35:45.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f376c0640 1 Processor -- start 2024-03-23T14:35:45.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f376c0640 1 -- start start 2024-03-23T14:35:45.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f376c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f300d1cb0 0x7f0f30154580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f376c0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f30161750 0x7f0f3009ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f376c0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30161b30 0x7f0f3006bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f376c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0f300a00d0 con 0x7f0f300d1cb0 2024-03-23T14:35:45.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f376c0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0f30154ac0 con 0x7f0f30161b30 2024-03-23T14:35:45.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f376c0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0f301f40a0 con 0x7f0f30161750 2024-03-23T14:35:45.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f3545c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f300d1cb0 0x7f0f30154580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f3545c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f300d1cb0 0x7f0f30154580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41602/0 (socket says 172.21.15.161:41602) 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.866+0000 7f0f3545c640 1 -- 172.21.15.161:0/1585100948 learned_addr learned my addr 172.21.15.161:0/1585100948 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f35c5d640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30161b30 0x7f0f3006bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f34c5b640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f30161750 0x7f0f3009ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f3545c640 1 -- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f30161750 msgr2=0x7f0f3009ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f3545c640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f30161750 0x7f0f3009ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f3545c640 1 -- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30161b30 msgr2=0x7f0f3006bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f3545c640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30161b30 0x7f0f3006bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f3545c640 1 -- 172.21.15.161:0/1585100948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f28009b30 con 0x7f0f300d1cb0 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f3545c640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f300d1cb0 0x7f0f30154580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00e9b0 tx=0x7f0f2c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f25ffb640 1 -- 172.21.15.161:0/1585100948 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c009670 con 0x7f0f300d1cb0 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f25ffb640 1 -- 172.21.15.161:0/1585100948 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f2c00c9a0 con 0x7f0f300d1cb0 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f25ffb640 1 -- 172.21.15.161:0/1585100948 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f2c00cd50 con 0x7f0f300d1cb0 2024-03-23T14:35:45.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f301f4340 con 0x7f0f300d1cb0 2024-03-23T14:35:45.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.867+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f301f4900 con 0x7f0f300d1cb0 2024-03-23T14:35:45.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.868+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f300d3790 con 0x7f0f300d1cb0 2024-03-23T14:35:45.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.869+0000 7f0f25ffb640 1 -- 172.21.15.161:0/1585100948 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0f2c013470 con 0x7f0f300d1cb0 2024-03-23T14:35:45.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.869+0000 7f0f25ffb640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0f1406b810 0x7f0f1406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:45.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.869+0000 7f0f25ffb640 1 -- 172.21.15.161:0/1585100948 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f0f2c08a2e0 con 0x7f0f300d1cb0 2024-03-23T14:35:45.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.869+0000 7f0f34c5b640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0f1406b810 0x7f0f1406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:45.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.870+0000 7f0f34c5b640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0f1406b810 0x7f0f1406dcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0f3015eef0 tx=0x7f0f20009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:45.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:45.873+0000 7f0f25ffb640 1 -- 172.21.15.161:0/1585100948 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0f2c054e40 con 0x7f0f300d1cb0 2024-03-23T14:35:46.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.045+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0f3004f550 con 0x7f0f300d1cb0 2024-03-23T14:35:46.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.045+0000 7f0f25ffb640 1 -- 172.21.15.161:0/1585100948 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f0f2c058500 con 0x7f0f300d1cb0 2024-03-23T14:35:46.041 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0f1406b810 msgr2=0x7f0f1406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0f1406b810 0x7f0f1406dcd0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0f3015eef0 tx=0x7f0f20009040 comp rx=0 tx=0).stop 2024-03-23T14:35:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f300d1cb0 msgr2=0x7f0f30154580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f300d1cb0 0x7f0f30154580 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0f2c00e9b0 tx=0x7f0f2c00ee70 comp rx=0 tx=0).stop 2024-03-23T14:35:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 shutdown_connections 2024-03-23T14:35:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0f1406b810 0x7f0f1406dcd0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0f30161b30 0x7f0f3006bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0f30161750 0x7f0f3009ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 --2- 172.21.15.161:0/1585100948 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0f300d1cb0 0x7f0f30154580 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 >> 172.21.15.161:0/1585100948 conn(0x7f0f3009d450 msgr2=0x7f0f3015fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 shutdown_connections 2024-03-23T14:35:46.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.047+0000 7f0f376c0640 1 -- 172.21.15.161:0/1585100948 wait complete. 2024-03-23T14:35:46.054 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:46.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.160+0000 7f6e4bd97640 1 Processor -- start 2024-03-23T14:35:46.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e4bd97640 1 -- start start 2024-03-23T14:35:46.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e4bd97640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6e440d0380 0x7f6e44165b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e4bd97640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e4415f480 con 0x7f6e441552f0 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e4bd97640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6e441556d0 con 0x7f6e440b2aa0 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e4bd97640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6e4406eee0 con 0x7f6e440d0380 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e4a334640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6e440d0380 0x7f6e44165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e49b33640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f6e441552f0 0x7f6e440b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32804/0 (socket says 172.21.15.161:32804) 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e49b33640 1 -- 172.21.15.161:0/3831785391 learned_addr learned my addr 172.21.15.161:0/3831785391 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.161+0000 7f6e4a334640 1 --2- 172.21.15.161:0/3831785391 >> v2:172.21.15.161:3300/0 conn(0x7f6e440d0380 0x7f6e44165b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41618/0 (socket says 172.21.15.161:41618) 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.162+0000 7f6e4a334640 1 -- 172.21.15.161:0/3831785391 >> v1:172.21.15.183:6789/0 conn(0x7f6e440b2aa0 legacy=0x7f6e440cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.162+0000 7f6e4a334640 1 -- 172.21.15.161:0/3831785391 >> v1:172.21.15.161:6789/0 conn(0x7f6e441552f0 legacy=0x7f6e440b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.162+0000 7f6e4a334640 1 -- 172.21.15.161:0/3831785391 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e4415e870 con 0x7f6e440d0380 2024-03-23T14:35:46.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.162+0000 7f6e48b31640 1 -- 172.21.15.161:0/3831785391 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2108667245 0 0) 0x7f6e4415f480 con 0x7f6e441552f0 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.162+0000 7f6e4a334640 1 --2- 172.21.15.161:0/3831785391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440d0380 0x7f6e44165b60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0097a0 tx=0x7f6e3c02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5592af604c2671c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e48b31640 1 -- 172.21.15.161:0/3831785391 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c030610 con 0x7f6e440d0380 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e48b31640 1 -- 172.21.15.161:0/3831785391 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e3c030770 con 0x7f6e440d0380 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e48b31640 1 -- 172.21.15.161:0/3831785391 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e3c030b00 con 0x7f6e440d0380 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e4bd97640 1 -- 172.21.15.161:0/3831785391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440d0380 msgr2=0x7f6e44165b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e4bd97640 1 --2- 172.21.15.161:0/3831785391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440d0380 0x7f6e44165b60 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6e3c0097a0 tx=0x7f6e3c02b960 comp rx=0 tx=0).stop 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e4bd97640 1 -- 172.21.15.161:0/3831785391 shutdown_connections 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e4bd97640 1 --2- 172.21.15.161:0/3831785391 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440d0380 0x7f6e44165b60 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e4bd97640 1 -- 172.21.15.161:0/3831785391 >> 172.21.15.161:0/3831785391 conn(0x7f6e4406b260 msgr2=0x7f6e4406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e4bd97640 1 -- 172.21.15.161:0/3831785391 shutdown_connections 2024-03-23T14:35:46.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.163+0000 7f6e4bd97640 1 -- 172.21.15.161:0/3831785391 wait complete. 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.164+0000 7f6e4bd97640 1 Processor -- start 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e4bd97640 1 -- start start 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e4bd97640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440b2aa0 0x7f6e44153660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e4bd97640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e440d0380 0x7f6e4415d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e4bd97640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e441552f0 0x7f6e44157e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e4bd97640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6e44156a80 con 0x7f6e440b2aa0 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e4bd97640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6e4415d2b0 con 0x7f6e440d0380 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e4bd97640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f6e441561e0 con 0x7f6e441552f0 2024-03-23T14:35:46.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e49b33640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440b2aa0 0x7f6e44153660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e49b33640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440b2aa0 0x7f6e44153660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41624/0 (socket says 172.21.15.161:41624) 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e49b33640 1 -- 172.21.15.161:0/556067389 learned_addr learned my addr 172.21.15.161:0/556067389 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e4a334640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e441552f0 0x7f6e44157e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.165+0000 7f6e49332640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e440d0380 0x7f6e4415d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e49b33640 1 -- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e441552f0 msgr2=0x7f6e44157e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e49b33640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e441552f0 0x7f6e44157e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e49b33640 1 -- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e440d0380 msgr2=0x7f6e4415d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e49b33640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e440d0380 0x7f6e4415d9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e49b33640 1 -- 172.21.15.161:0/556067389 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e3c009400 con 0x7f6e440b2aa0 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e49332640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e440d0380 0x7f6e4415d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e49b33640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440b2aa0 0x7f6e44153660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c8c0 tx=0x7f6e4000cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e3a7fc640 1 -- 172.21.15.161:0/556067389 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4000a640 con 0x7f6e440b2aa0 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e3a7fc640 1 -- 172.21.15.161:0/556067389 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6e4000a7a0 con 0x7f6e440b2aa0 2024-03-23T14:35:46.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e441564c0 con 0x7f6e440b2aa0 2024-03-23T14:35:46.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e3a7fc640 1 -- 172.21.15.161:0/556067389 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4000ab90 con 0x7f6e440b2aa0 2024-03-23T14:35:46.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.166+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e44156850 con 0x7f6e440b2aa0 2024-03-23T14:35:46.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.167+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e440b4d10 con 0x7f6e440b2aa0 2024-03-23T14:35:46.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.167+0000 7f6e3a7fc640 1 -- 172.21.15.161:0/556067389 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6e4000acf0 con 0x7f6e440b2aa0 2024-03-23T14:35:46.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.168+0000 7f6e3a7fc640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6e2806b810 0x7f6e2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.168+0000 7f6e3a7fc640 1 -- 172.21.15.161:0/556067389 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f6e40089930 con 0x7f6e440b2aa0 2024-03-23T14:35:46.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.168+0000 7f6e49332640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6e2806b810 0x7f6e2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.168+0000 7f6e49332640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6e2806b810 0x7f6e2806dcd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6e34007fd0 tx=0x7f6e34009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:46.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.172+0000 7f6e3a7fc640 1 -- 172.21.15.161:0/556067389 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6e40054490 con 0x7f6e440b2aa0 2024-03-23T14:35:46.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.344+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6e441526e0 con 0x7f6e440b2aa0 2024-03-23T14:35:46.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.344+0000 7f6e3a7fc640 1 -- 172.21.15.161:0/556067389 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f6e40018020 con 0x7f6e440b2aa0 2024-03-23T14:35:46.343 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6e2806b810 msgr2=0x7f6e2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6e2806b810 0x7f6e2806dcd0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6e34007fd0 tx=0x7f6e34009040 comp rx=0 tx=0).stop 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440b2aa0 msgr2=0x7f6e44153660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440b2aa0 0x7f6e44153660 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000c8c0 tx=0x7f6e4000cd80 comp rx=0 tx=0).stop 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 shutdown_connections 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6e2806b810 0x7f6e2806dcd0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6e441552f0 0x7f6e44157e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6e440d0380 0x7f6e4415d9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 --2- 172.21.15.161:0/556067389 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6e440b2aa0 0x7f6e44153660 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 >> 172.21.15.161:0/556067389 conn(0x7f6e4406b260 msgr2=0x7f6e44154470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 shutdown_connections 2024-03-23T14:35:46.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.346+0000 7f6e4bd97640 1 -- 172.21.15.161:0/556067389 wait complete. 2024-03-23T14:35:46.353 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:35:46.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.457+0000 7f8700f9a640 1 Processor -- start 2024-03-23T14:35:46.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.457+0000 7f8700f9a640 1 -- start start 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.458+0000 7f8700f9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f86fc15e400 0x7f86fc0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.458+0000 7f8700f9a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f86fc0cf9d0 0x7f86fc0d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.458+0000 7f8700f9a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86fc0d0460 con 0x7f86fc0d1470 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.458+0000 7f8700f9a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f86fc0d1050 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.458+0000 7f8700f9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f86fc0d06b0 con 0x7f86fc15e400 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86f9d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f86fc0cf9d0 0x7f86fc0d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86f9d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f86fc0cf9d0 0x7f86fc0d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41628/0 (socket says 172.21.15.161:41628) 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86fa59c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f86fc15e400 0x7f86fc0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86f9d9b640 1 -- 172.21.15.161:0/1896258294 learned_addr learned my addr 172.21.15.161:0/1896258294 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86f9d9b640 1 -- 172.21.15.161:0/1896258294 >> v2:172.21.15.183:3300/0 conn(0x7f86fc15e400 msgr2=0x7f86fc0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86f9d9b640 1 --2- 172.21.15.161:0/1896258294 >> v2:172.21.15.183:3300/0 conn(0x7f86fc15e400 0x7f86fc0cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86f9d9b640 1 -- 172.21.15.161:0/1896258294 >> v1:172.21.15.183:6789/0 conn(0x7f86fc0d1470 legacy=0x7f86fc0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86f9d9b640 1 -- 172.21.15.161:0/1896258294 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86fc0d0990 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.459+0000 7f86f9d9b640 1 --2- 172.21.15.161:0/1896258294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc0cf9d0 0x7f86fc0d2430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f86fc08e230 tx=0x7f86f002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1595f52379ca5e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f86f959a640 1 -- 172.21.15.161:0/1896258294 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f0030860 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f86f959a640 1 -- 172.21.15.161:0/1896258294 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86f00309c0 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f86f959a640 1 -- 172.21.15.161:0/1896258294 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86f0030d50 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f8700f9a640 1 -- 172.21.15.161:0/1896258294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc0cf9d0 msgr2=0x7f86fc0d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f8700f9a640 1 --2- 172.21.15.161:0/1896258294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc0cf9d0 0x7f86fc0d2430 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f86fc08e230 tx=0x7f86f002b9a0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f8700f9a640 1 -- 172.21.15.161:0/1896258294 shutdown_connections 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f8700f9a640 1 --2- 172.21.15.161:0/1896258294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc0cf9d0 0x7f86fc0d2430 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f8700f9a640 1 --2- 172.21.15.161:0/1896258294 >> v2:172.21.15.183:3300/0 conn(0x7f86fc15e400 0x7f86fc0cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f8700f9a640 1 -- 172.21.15.161:0/1896258294 >> 172.21.15.161:0/1896258294 conn(0x7f86fc09d190 msgr2=0x7f86fc0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:46.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f8700f9a640 1 -- 172.21.15.161:0/1896258294 shutdown_connections 2024-03-23T14:35:46.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.460+0000 7f8700f9a640 1 -- 172.21.15.161:0/1896258294 wait complete. 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.462+0000 7f8700f9a640 1 Processor -- start 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.462+0000 7f8700f9a640 1 -- start start 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.462+0000 7f8700f9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86fc0cf9d0 0x7f86fc1f3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.462+0000 7f8700f9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86fc0d1470 0x7f86fc06cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.462+0000 7f8700f9a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc15e400 0x7f86fc06d100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.462+0000 7f8700f9a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc1f4000 con 0x7f86fc15e400 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.462+0000 7f8700f9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f86fc0cab80 con 0x7f86fc0d1470 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.462+0000 7f8700f9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f86fc0ca8a0 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86fc0cf9d0 0x7f86fc1f3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc15e400 0x7f86fc06d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86fc0cf9d0 0x7f86fc1f3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:52098/0 (socket says 172.21.15.161:52098) 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86f9d9b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86fc0d1470 0x7f86fc06cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 -- 172.21.15.161:0/3632819500 learned_addr learned my addr 172.21.15.161:0/3632819500 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 -- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86fc0d1470 msgr2=0x7f86fc06cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86fc0d1470 0x7f86fc06cbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 -- 172.21.15.161:0/3632819500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc15e400 msgr2=0x7f86fc06d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc15e400 0x7f86fc06d100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 -- 172.21.15.161:0/3632819500 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86f00094d0 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fad9d640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc15e400 0x7f86fc06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:46.461 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86fa59c640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86fc0cf9d0 0x7f86fc1f3ac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00e870 tx=0x7f86ec00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:46.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86eaffd640 1 -- 172.21.15.161:0/3632819500 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec0094f0 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86eaffd640 1 -- 172.21.15.161:0/3632819500 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86ec00c940 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86fc1482d0 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f86eaffd640 1 -- 172.21.15.161:0/3632819500 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ec00ccf0 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.463+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f86fc148890 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.464+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86fc0a1e00 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.465+0000 7f86eaffd640 1 -- 172.21.15.161:0/3632819500 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f86ec013470 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.465+0000 7f86eaffd640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f86dc06b810 0x7f86dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:46.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.465+0000 7f86eaffd640 1 -- 172.21.15.161:0/3632819500 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f86ec08a1e0 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.465+0000 7f86f9d9b640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f86dc06b810 0x7f86dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:46.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.465+0000 7f86f9d9b640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f86dc06b810 0x7f86dc06dcd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f86f0036000 tx=0x7f86f0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:46.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.470+0000 7f86eaffd640 1 -- 172.21.15.161:0/3632819500 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f86ec054d40 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.651+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f86fc04f550 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.652+0000 7f86eaffd640 1 -- 172.21.15.161:0/3632819500 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f86ec058400 con 0x7f86fc0cf9d0 2024-03-23T14:35:46.648 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.653+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f86dc06b810 msgr2=0x7f86dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f86dc06b810 0x7f86dc06dcd0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f86f0036000 tx=0x7f86f0008030 comp rx=0 tx=0).stop 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86fc0cf9d0 msgr2=0x7f86fc1f3ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86fc0cf9d0 0x7f86fc1f3ac0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f86ec00e870 tx=0x7f86ec00ed30 comp rx=0 tx=0).stop 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 shutdown_connections 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f86dc06b810 0x7f86dc06dcd0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86fc15e400 0x7f86fc06d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86fc0d1470 0x7f86fc06cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 --2- 172.21.15.161:0/3632819500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86fc0cf9d0 0x7f86fc1f3ac0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 >> 172.21.15.161:0/3632819500 conn(0x7f86fc09d190 msgr2=0x7f86fc161d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:46.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 shutdown_connections 2024-03-23T14:35:46.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:46.654+0000 7f8700f9a640 1 -- 172.21.15.161:0/3632819500 wait complete. 2024-03-23T14:35:46.660 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi183.front.sepia.ceph.com... 2024-03-23T14:35:46.660 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:46.661 DEBUG:teuthology.orchestra.run.smithi183:> sudo fusermount -u /home/ubuntu/cephtest/mnt.1 2024-03-23T14:35:46.688 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:35:46.688 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:35:46.688 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:46.688 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T14:35:46.840 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:35:46.840 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:46.856 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:35:46.856 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:46.912 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:47.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.058+0000 7fd5a3b6c640 1 Processor -- start 2024-03-23T14:35:47.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.058+0000 7fd5a3b6c640 1 -- start start 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a3b6c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd59c1607d0 0x7fd59c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a3b6c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd59c0cf4f0 con 0x7fd59c145080 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a3b6c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd59c0cef30 con 0x7fd59c1601f0 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a3b6c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd59c071010 con 0x7fd59c1607d0 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a2109640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd59c1607d0 0x7fd59c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a2109640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd59c1607d0 0x7fd59c072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41644/0 (socket says 172.21.15.161:41644) 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a2109640 1 -- 172.21.15.161:0/2383525222 learned_addr learned my addr 172.21.15.161:0/2383525222 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a1908640 1 --1- 172.21.15.161:0/2383525222 >> v1:172.21.15.161:6789/0 conn(0x7fd59c145080 0x7fd59c15fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32812/0 (socket says 172.21.15.161:32812) 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a2109640 1 -- 172.21.15.161:0/2383525222 >> v1:172.21.15.183:6789/0 conn(0x7fd59c1601f0 legacy=0x7fd59c15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a2109640 1 -- 172.21.15.161:0/2383525222 >> v1:172.21.15.161:6789/0 conn(0x7fd59c145080 legacy=0x7fd59c15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a0906640 1 -- 172.21.15.161:0/2383525222 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3420827323 0 0) 0x7fd59c0cef30 con 0x7fd59c1601f0 2024-03-23T14:35:47.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.059+0000 7fd5a2109640 1 -- 172.21.15.161:0/2383525222 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd59c0cf210 con 0x7fd59c1607d0 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.060+0000 7fd5a2109640 1 --2- 172.21.15.161:0/2383525222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c1607d0 0x7fd59c072a00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd594009e50 tx=0x7fd59402bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=751c7c1bedbff477 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.060+0000 7fd5a0906640 1 -- 172.21.15.161:0/2383525222 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59402c8d0 con 0x7fd59c1607d0 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.060+0000 7fd5a0906640 1 -- 172.21.15.161:0/2383525222 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd59402ca30 con 0x7fd59c1607d0 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.060+0000 7fd5a0906640 1 -- 172.21.15.161:0/2383525222 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd59402cdc0 con 0x7fd59c1607d0 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.060+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/2383525222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c1607d0 msgr2=0x7fd59c072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.060+0000 7fd5a3b6c640 1 --2- 172.21.15.161:0/2383525222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c1607d0 0x7fd59c072a00 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd594009e50 tx=0x7fd59402bdb0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.061+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/2383525222 shutdown_connections 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.061+0000 7fd5a3b6c640 1 --2- 172.21.15.161:0/2383525222 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c1607d0 0x7fd59c072a00 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.061+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/2383525222 >> 172.21.15.161:0/2383525222 conn(0x7fd59c0b17f0 msgr2=0x7fd59c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:47.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.061+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/2383525222 shutdown_connections 2024-03-23T14:35:47.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.061+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/2383525222 wait complete. 2024-03-23T14:35:47.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a3b6c640 1 Processor -- start 2024-03-23T14:35:47.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a3b6c640 1 -- start start 2024-03-23T14:35:47.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a3b6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c145080 0x7fd59c0b3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a3b6c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c1601f0 0x7fd59c15e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a3b6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c1607d0 0x7fd59c15ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a3b6c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c13afb0 con 0x7fd59c145080 2024-03-23T14:35:47.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a3b6c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd59c1f41b0 con 0x7fd59c1601f0 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a3b6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd59c0b48d0 con 0x7fd59c1607d0 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a1908640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c145080 0x7fd59c0b3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.062+0000 7fd5a1908640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c145080 0x7fd59c0b3e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41660/0 (socket says 172.21.15.161:41660) 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a1908640 1 -- 172.21.15.161:0/3856163736 learned_addr learned my addr 172.21.15.161:0/3856163736 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a2109640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c1607d0 0x7fd59c15ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a1107640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c1601f0 0x7fd59c15e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a1908640 1 -- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c1607d0 msgr2=0x7fd59c15ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a1908640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c1607d0 0x7fd59c15ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a1908640 1 -- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c1601f0 msgr2=0x7fd59c15e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a1908640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c1601f0 0x7fd59c15e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a1908640 1 -- 172.21.15.161:0/3856163736 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd594009b30 con 0x7fd59c145080 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a1908640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c145080 0x7fd59c0b3e30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd59800c780 tx=0x7fd59800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5927fc640 1 -- 172.21.15.161:0/3856163736 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5980072d0 con 0x7fd59c145080 2024-03-23T14:35:47.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5927fc640 1 -- 172.21.15.161:0/3856163736 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd598007430 con 0x7fd59c145080 2024-03-23T14:35:47.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd59c0b2bf0 con 0x7fd59c145080 2024-03-23T14:35:47.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5927fc640 1 -- 172.21.15.161:0/3856163736 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5980047e0 con 0x7fd59c145080 2024-03-23T14:35:47.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.063+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd59c0b31b0 con 0x7fd59c145080 2024-03-23T14:35:47.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.064+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd59c06cb70 con 0x7fd59c145080 2024-03-23T14:35:47.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.064+0000 7fd5927fc640 1 -- 172.21.15.161:0/3856163736 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd598004940 con 0x7fd59c145080 2024-03-23T14:35:47.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.065+0000 7fd5927fc640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd58006b810 0x7fd58006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.065+0000 7fd5927fc640 1 -- 172.21.15.161:0/3856163736 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd5980897f0 con 0x7fd59c145080 2024-03-23T14:35:47.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.065+0000 7fd5a1107640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd58006b810 0x7fd58006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.066+0000 7fd5a1107640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd58006b810 0x7fd58006dcd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd59c13eaa0 tx=0x7fd58c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.069+0000 7fd5927fc640 1 -- 172.21.15.161:0/3856163736 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd5980542c0 con 0x7fd59c145080 2024-03-23T14:35:47.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.242+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd59c0735e0 con 0x7fd59c145080 2024-03-23T14:35:47.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.244+0000 7fd5927fc640 1 -- 172.21.15.161:0/3856163736 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7fd598057980 con 0x7fd59c145080 2024-03-23T14:35:47.240 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:47.240 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd58006b810 msgr2=0x7fd58006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd58006b810 0x7fd58006dcd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd59c13eaa0 tx=0x7fd58c009040 comp rx=0 tx=0).stop 2024-03-23T14:35:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c145080 msgr2=0x7fd59c0b3e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c145080 0x7fd59c0b3e30 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fd59800c780 tx=0x7fd59800cc40 comp rx=0 tx=0).stop 2024-03-23T14:35:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 shutdown_connections 2024-03-23T14:35:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd58006b810 0x7fd58006dcd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd59c1607d0 0x7fd59c15ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd59c1601f0 0x7fd59c15e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 --2- 172.21.15.161:0/3856163736 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd59c145080 0x7fd59c0b3e30 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 >> 172.21.15.161:0/3856163736 conn(0x7fd59c0b17f0 msgr2=0x7fd59c13de20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 shutdown_connections 2024-03-23T14:35:47.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.246+0000 7fd5a3b6c640 1 -- 172.21.15.161:0/3856163736 wait complete. 2024-03-23T14:35:47.253 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:47.253 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:47.356 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.361+0000 7f9715c15640 1 Processor -- start 2024-03-23T14:35:47.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.361+0000 7f9715c15640 1 -- start start 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.361+0000 7f9715c15640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f97100d0380 0x7f9710165b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.361+0000 7f9715c15640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f971015f480 con 0x7f97100b2aa0 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.361+0000 7f9715c15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97101556d0 con 0x7f97101552f0 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.361+0000 7f9715c15640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f971006eee0 con 0x7f97100d0380 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f9706ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f97100b2aa0 0x7f97100cfc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32820/0 (socket says 172.21.15.161:32820) 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f9706ffd640 1 -- 172.21.15.161:0/839962727 learned_addr learned my addr 172.21.15.161:0/839962727 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f970ffff640 1 --2- 172.21.15.161:0/839962727 >> v2:172.21.15.183:3300/0 conn(0x7f97100d0380 0x7f9710165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f970effd640 1 -- 172.21.15.161:0/839962727 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516250038 0 0) 0x7f971015f480 con 0x7f97100b2aa0 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f970effd640 1 -- 172.21.15.161:0/839962727 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96fc003520 con 0x7f97100b2aa0 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f970ffff640 1 -- 172.21.15.161:0/839962727 >> v1:172.21.15.183:6789/0 conn(0x7f97101552f0 legacy=0x7f97100b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f970effd640 1 -- 172.21.15.161:0/839962727 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1293298352 0 0) 0x7f97101556d0 con 0x7f97101552f0 2024-03-23T14:35:47.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f970ffff640 1 -- 172.21.15.161:0/839962727 >> v1:172.21.15.161:6789/0 conn(0x7f97100b2aa0 legacy=0x7f97100cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.362+0000 7f970ffff640 1 -- 172.21.15.161:0/839962727 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f971015e870 con 0x7f97100d0380 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.363+0000 7f970ffff640 1 --2- 172.21.15.161:0/839962727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 0x7f9710165b60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9708009d90 tx=0x7f970802bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8a7f858f4a3721f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.363+0000 7f970effd640 1 -- 172.21.15.161:0/839962727 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f970802cd00 con 0x7f97100d0380 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.363+0000 7f970effd640 1 -- 172.21.15.161:0/839962727 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f970802ce60 con 0x7f97100d0380 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.363+0000 7f970effd640 1 -- 172.21.15.161:0/839962727 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9708034660 con 0x7f97100d0380 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.363+0000 7f9715c15640 1 -- 172.21.15.161:0/839962727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 msgr2=0x7f9710165b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.363+0000 7f9715c15640 1 --2- 172.21.15.161:0/839962727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 0x7f9710165b60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f9708009d90 tx=0x7f970802bf50 comp rx=0 tx=0).stop 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.363+0000 7f9715c15640 1 -- 172.21.15.161:0/839962727 shutdown_connections 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.363+0000 7f9715c15640 1 --2- 172.21.15.161:0/839962727 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 0x7f9710165b60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.364+0000 7f9715c15640 1 -- 172.21.15.161:0/839962727 >> 172.21.15.161:0/839962727 conn(0x7f971006b260 msgr2=0x7f971006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:47.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.364+0000 7f9715c15640 1 -- 172.21.15.161:0/839962727 shutdown_connections 2024-03-23T14:35:47.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.364+0000 7f9715c15640 1 -- 172.21.15.161:0/839962727 wait complete. 2024-03-23T14:35:47.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.365+0000 7f9715c15640 1 Processor -- start 2024-03-23T14:35:47.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.365+0000 7f9715c15640 1 -- start start 2024-03-23T14:35:47.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9715c15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97100b2aa0 0x7f97101f5fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9715c15640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 0x7f971013cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9715c15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97101552f0 0x7f9710071bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9715c15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f97101571d0 con 0x7f97100b2aa0 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9715c15640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9710157fc0 con 0x7f97100d0380 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9715c15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f97101f6500 con 0x7f97101552f0 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97100b2aa0 0x7f97101f5fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97100b2aa0 0x7f97101f5fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41670/0 (socket says 172.21.15.161:41670) 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 -- 172.21.15.161:0/288279833 learned_addr learned my addr 172.21.15.161:0/288279833 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9706ffd640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 0x7f971013cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970ffff640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97101552f0 0x7f9710071bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 -- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97101552f0 msgr2=0x7f9710071bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97101552f0 0x7f9710071bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 -- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 msgr2=0x7f971013cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 0x7f971013cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 -- 172.21.15.161:0/288279833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9708009a70 con 0x7f97100b2aa0 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970f7fe640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97100b2aa0 0x7f97101f5fc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f970000c4b0 tx=0x7f970000c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970cff9640 1 -- 172.21.15.161:0/288279833 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9700005480 con 0x7f97100b2aa0 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970cff9640 1 -- 172.21.15.161:0/288279833 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97000035f0 con 0x7f97100b2aa0 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97101573b0 con 0x7f97100b2aa0 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f970cff9640 1 -- 172.21.15.161:0/288279833 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97000039e0 con 0x7f97100b2aa0 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.366+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9710157970 con 0x7f97100b2aa0 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.367+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f971004f4e0 con 0x7f97100b2aa0 2024-03-23T14:35:47.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.368+0000 7f970cff9640 1 -- 172.21.15.161:0/288279833 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9700003b40 con 0x7f97100b2aa0 2024-03-23T14:35:47.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.368+0000 7f970cff9640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96f006b810 0x7f96f006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.368+0000 7f970cff9640 1 -- 172.21.15.161:0/288279833 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f9700089750 con 0x7f97100b2aa0 2024-03-23T14:35:47.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.368+0000 7f9706ffd640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96f006b810 0x7f96f006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.369+0000 7f9706ffd640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96f006b810 0x7f96f006dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9710158450 tx=0x7f96f8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.373+0000 7f970cff9640 1 -- 172.21.15.161:0/288279833 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9700054220 con 0x7f97100b2aa0 2024-03-23T14:35:47.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.546+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97100cb980 con 0x7f97100b2aa0 2024-03-23T14:35:47.542 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.546+0000 7f970cff9640 1 -- 172.21.15.161:0/288279833 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f97000578e0 con 0x7f97100b2aa0 2024-03-23T14:35:47.542 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:47.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96f006b810 msgr2=0x7f96f006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96f006b810 0x7f96f006dcd0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f9710158450 tx=0x7f96f8009040 comp rx=0 tx=0).stop 2024-03-23T14:35:47.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97100b2aa0 msgr2=0x7f97101f5fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97100b2aa0 0x7f97101f5fc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f970000c4b0 tx=0x7f970000c970 comp rx=0 tx=0).stop 2024-03-23T14:35:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 shutdown_connections 2024-03-23T14:35:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96f006b810 0x7f96f006dcd0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97101552f0 0x7f9710071bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97100d0380 0x7f971013cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 --2- 172.21.15.161:0/288279833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97100b2aa0 0x7f97101f5fc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 >> 172.21.15.161:0/288279833 conn(0x7f971006b260 msgr2=0x7f9710153340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 shutdown_connections 2024-03-23T14:35:47.545 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.548+0000 7f9715c15640 1 -- 172.21.15.161:0/288279833 wait complete. 2024-03-23T14:35:47.555 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:47.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.663+0000 7f3dd991e640 1 Processor -- start 2024-03-23T14:35:47.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.663+0000 7f3dd991e640 1 -- start start 2024-03-23T14:35:47.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd991e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd415de70 0x7f3dd415e250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd991e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3dd40d2130 0x7f3dd406c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd991e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3dd40b3710 con 0x7f3dd415e790 2024-03-23T14:35:47.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd991e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3dd415ecd0 con 0x7f3dd40d2130 2024-03-23T14:35:47.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd991e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd415efb0 con 0x7f3dd415de70 2024-03-23T14:35:47.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3dd40d2130 0x7f3dd406c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd2ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd415de70 0x7f3dd415e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd2ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd415de70 0x7f3dd415e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:52118/0 (socket says 172.21.15.161:52118) 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3dd40d2130 0x7f3dd406c570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41674/0 (socket says 172.21.15.161:41674) 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd27fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3dd415e790 0x7f3dd40d1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:44260/0 (socket says 172.21.15.161:44260) 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd2ffd640 1 -- 172.21.15.161:0/2826755843 learned_addr learned my addr 172.21.15.161:0/2826755843 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd2ffd640 1 -- 172.21.15.161:0/2826755843 >> v2:172.21.15.161:3300/0 conn(0x7f3dd40d2130 msgr2=0x7f3dd406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd2ffd640 1 --2- 172.21.15.161:0/2826755843 >> v2:172.21.15.161:3300/0 conn(0x7f3dd40d2130 0x7f3dd406c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd2ffd640 1 -- 172.21.15.161:0/2826755843 >> v1:172.21.15.183:6789/0 conn(0x7f3dd415e790 legacy=0x7f3dd40d1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.664+0000 7f3dd2ffd640 1 -- 172.21.15.161:0/2826755843 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dd415f290 con 0x7f3dd415de70 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.665+0000 7f3dd2ffd640 1 --2- 172.21.15.161:0/2826755843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd415de70 0x7f3dd415e250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009870 tx=0x7f3dc402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b93239e3bc273310 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.665+0000 7f3dd1ffb640 1 -- 172.21.15.161:0/2826755843 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4030840 con 0x7f3dd415de70 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.665+0000 7f3dd1ffb640 1 -- 172.21.15.161:0/2826755843 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc40309a0 con 0x7f3dd415de70 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.665+0000 7f3dd1ffb640 1 -- 172.21.15.161:0/2826755843 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc4030d30 con 0x7f3dd415de70 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.666+0000 7f3dd991e640 1 -- 172.21.15.161:0/2826755843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd415de70 msgr2=0x7f3dd415e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.666+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2826755843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd415de70 0x7f3dd415e250 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4009870 tx=0x7f3dc402b980 comp rx=0 tx=0).stop 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.666+0000 7f3dd991e640 1 -- 172.21.15.161:0/2826755843 shutdown_connections 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.666+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2826755843 >> v2:172.21.15.161:3300/0 conn(0x7f3dd40d2130 0x7f3dd406c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.666+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2826755843 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd415de70 0x7f3dd415e250 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.666+0000 7f3dd991e640 1 -- 172.21.15.161:0/2826755843 >> 172.21.15.161:0/2826755843 conn(0x7f3dd409aa40 msgr2=0x7f3dd409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.666+0000 7f3dd991e640 1 -- 172.21.15.161:0/2826755843 shutdown_connections 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.666+0000 7f3dd991e640 1 -- 172.21.15.161:0/2826755843 wait complete. 2024-03-23T14:35:47.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.667+0000 7f3dd991e640 1 Processor -- start 2024-03-23T14:35:47.664 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.667+0000 7f3dd991e640 1 -- start start 2024-03-23T14:35:47.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd991e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd40d2130 0x7f3dd40b11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd991e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dd415de70 0x7f3dd4148510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd991e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd415e790 0x7f3dd40cea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd991e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd4148a50 con 0x7f3dd415de70 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd991e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3dd414bc90 con 0x7f3dd415e790 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd991e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3dd40b0420 con 0x7f3dd40d2130 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dd415de70 0x7f3dd4148510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dd415de70 0x7f3dd4148510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41686/0 (socket says 172.21.15.161:41686) 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 -- 172.21.15.161:0/2635972848 learned_addr learned my addr 172.21.15.161:0/2635972848 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd2ffd640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd40d2130 0x7f3dd40b11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd37fe640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd415e790 0x7f3dd40cea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 -- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd40d2130 msgr2=0x7f3dd40b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd40d2130 0x7f3dd40b11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 -- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd415e790 msgr2=0x7f3dd40cea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd415e790 0x7f3dd40cea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 -- 172.21.15.161:0/2635972848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3dc40094d0 con 0x7f3dd415de70 2024-03-23T14:35:47.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.668+0000 7f3dd27fc640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dd415de70 0x7f3dd4148510 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3dc800c3e0 tx=0x7f3dc800c8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.669+0000 7f3dbbfff640 1 -- 172.21.15.161:0/2635972848 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc80074e0 con 0x7f3dd415de70 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.669+0000 7f3dbbfff640 1 -- 172.21.15.161:0/2635972848 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3dc8007640 con 0x7f3dd415de70 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.669+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3dd40b0700 con 0x7f3dd415de70 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.669+0000 7f3dbbfff640 1 -- 172.21.15.161:0/2635972848 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3dc80049e0 con 0x7f3dd415de70 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.669+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3dd40b0cc0 con 0x7f3dd415de70 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.670+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3dd409ff90 con 0x7f3dd415de70 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.670+0000 7f3dbbfff640 1 -- 172.21.15.161:0/2635972848 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3dc8004b40 con 0x7f3dd415de70 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.670+0000 7f3dbbfff640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3dc006b840 0x7f3dc006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.671+0000 7f3dd2ffd640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3dc006b840 0x7f3dc006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.671+0000 7f3dbbfff640 1 -- 172.21.15.161:0/2635972848 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3dc8089ac0 con 0x7f3dd415de70 2024-03-23T14:35:47.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.671+0000 7f3dd2ffd640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3dc006b840 0x7f3dc006dd00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4036000 tx=0x7f3dc4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.675+0000 7f3dbbfff640 1 -- 172.21.15.161:0/2635972848 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3dc8054510 con 0x7f3dd415de70 2024-03-23T14:35:47.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.846+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3dd404f550 con 0x7f3dd415de70 2024-03-23T14:35:47.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.847+0000 7f3dbbfff640 1 -- 172.21.15.161:0/2635972848 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f3dc8057bd0 con 0x7f3dd415de70 2024-03-23T14:35:47.843 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3dc006b840 msgr2=0x7f3dc006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3dc006b840 0x7f3dc006dd00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3dc4036000 tx=0x7f3dc4008030 comp rx=0 tx=0).stop 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dd415de70 msgr2=0x7f3dd4148510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dd415de70 0x7f3dd4148510 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f3dc800c3e0 tx=0x7f3dc800c8a0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 shutdown_connections 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3dc006b840 0x7f3dc006dd00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3dd415e790 0x7f3dd40cea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3dd415de70 0x7f3dd4148510 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 --2- 172.21.15.161:0/2635972848 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3dd40d2130 0x7f3dd40b11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 >> 172.21.15.161:0/2635972848 conn(0x7f3dd409aa40 msgr2=0x7f3dd40b2aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 shutdown_connections 2024-03-23T14:35:47.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.849+0000 7f3dd991e640 1 -- 172.21.15.161:0/2635972848 wait complete. 2024-03-23T14:35:47.856 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:47.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.964+0000 7f19d2ac9640 1 Processor -- start 2024-03-23T14:35:47.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.964+0000 7f19d2ac9640 1 -- start start 2024-03-23T14:35:47.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.964+0000 7f19d2ac9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f19cc1552f0 0x7f19cc0b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.964+0000 7f19d2ac9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc0d0380 0x7f19cc06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.964+0000 7f19d2ac9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f19cc15f480 con 0x7f19cc0b2aa0 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.964+0000 7f19d2ac9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f19cc1556d0 con 0x7f19cc1552f0 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.964+0000 7f19d2ac9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f19cc15e870 con 0x7f19cc0d0380 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19d0865640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f19cc1552f0 0x7f19cc0b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19d1066640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc0d0380 0x7f19cc06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19cbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f19cc0b2aa0 0x7f19cc0cfc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32830/0 (socket says 172.21.15.161:32830) 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19d0865640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f19cc1552f0 0x7f19cc0b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41690/0 (socket says 172.21.15.161:41690) 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19cbfff640 1 -- 172.21.15.161:0/3212637897 learned_addr learned my addr 172.21.15.161:0/3212637897 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:47.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19d0865640 1 -- 172.21.15.161:0/3212637897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc0d0380 msgr2=0x7f19cc06ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19d0865640 1 --2- 172.21.15.161:0/3212637897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc0d0380 0x7f19cc06ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19d0865640 1 -- 172.21.15.161:0/3212637897 >> v1:172.21.15.161:6789/0 conn(0x7f19cc0b2aa0 legacy=0x7f19cc0cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19cb7fe640 1 -- 172.21.15.161:0/3212637897 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1261117855 0 0) 0x7f19cc15f480 con 0x7f19cc0b2aa0 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.965+0000 7f19d0865640 1 -- 172.21.15.161:0/3212637897 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19cc15eb50 con 0x7f19cc1552f0 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d0865640 1 --2- 172.21.15.161:0/3212637897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc1552f0 0x7f19cc0b3d10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f19c00097a0 tx=0x7f19c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf6314872628693d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19cb7fe640 1 -- 172.21.15.161:0/3212637897 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0031830 con 0x7f19cc1552f0 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19cb7fe640 1 -- 172.21.15.161:0/3212637897 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19c0031990 con 0x7f19cc1552f0 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19cb7fe640 1 -- 172.21.15.161:0/3212637897 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19c0031d20 con 0x7f19cc1552f0 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d2ac9640 1 -- 172.21.15.161:0/3212637897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc1552f0 msgr2=0x7f19cc0b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/3212637897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc1552f0 0x7f19cc0b3d10 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f19c00097a0 tx=0x7f19c002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d2ac9640 1 -- 172.21.15.161:0/3212637897 shutdown_connections 2024-03-23T14:35:47.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/3212637897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc0d0380 0x7f19cc06ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/3212637897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc1552f0 0x7f19cc0b3d10 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d2ac9640 1 -- 172.21.15.161:0/3212637897 >> 172.21.15.161:0/3212637897 conn(0x7f19cc06b260 msgr2=0x7f19cc06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:47.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d2ac9640 1 -- 172.21.15.161:0/3212637897 shutdown_connections 2024-03-23T14:35:47.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.966+0000 7f19d2ac9640 1 -- 172.21.15.161:0/3212637897 wait complete. 2024-03-23T14:35:47.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.967+0000 7f19d2ac9640 1 Processor -- start 2024-03-23T14:35:47.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.967+0000 7f19d2ac9640 1 -- start start 2024-03-23T14:35:47.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19d2ac9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19cc0b2aa0 0x7f19cc15cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19d2ac9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc0d0380 0x7f19cc1445a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19d2ac9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc1552f0 0x7f19cc0cd660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19d2ac9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc15e5b0 con 0x7f19cc0d0380 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19d2ac9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f19cc15d510 con 0x7f19cc0b2aa0 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19d2ac9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f19cc15d7f0 con 0x7f19cc1552f0 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19cbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc0d0380 0x7f19cc1445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19cbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc0d0380 0x7f19cc1445a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41702/0 (socket says 172.21.15.161:41702) 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19cbfff640 1 -- 172.21.15.161:0/4277450286 learned_addr learned my addr 172.21.15.161:0/4277450286 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:47.968 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19d1066640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc1552f0 0x7f19cc0cd660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.968+0000 7f19d0865640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19cc0b2aa0 0x7f19cc15cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19cbfff640 1 -- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc1552f0 msgr2=0x7f19cc0cd660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19cbfff640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc1552f0 0x7f19cc0cd660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19cbfff640 1 -- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19cc0b2aa0 msgr2=0x7f19cc15cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19cbfff640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19cc0b2aa0 0x7f19cc15cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19cbfff640 1 -- 172.21.15.161:0/4277450286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19c0009400 con 0x7f19cc0d0380 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19cbfff640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc0d0380 0x7f19cc1445a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f19b800c4a0 tx=0x7f19b800c960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19c97fa640 1 -- 172.21.15.161:0/4277450286 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b80074e0 con 0x7f19cc0d0380 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19c97fa640 1 -- 172.21.15.161:0/4277450286 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f19b8007640 con 0x7f19cc0d0380 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19cc15dad0 con 0x7f19cc0d0380 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19c97fa640 1 -- 172.21.15.161:0/4277450286 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19b8004a00 con 0x7f19cc0d0380 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.969+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19cc15e090 con 0x7f19cc0d0380 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.970+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19cc15e280 con 0x7f19cc0d0380 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.970+0000 7f19c97fa640 1 -- 172.21.15.161:0/4277450286 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f19b8004b60 con 0x7f19cc0d0380 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.971+0000 7f19c97fa640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f19b006b810 0x7f19b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:47.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.971+0000 7f19c97fa640 1 -- 172.21.15.161:0/4277450286 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f19b8089620 con 0x7f19cc0d0380 2024-03-23T14:35:47.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.971+0000 7f19d0865640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f19b006b810 0x7f19b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:47.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.972+0000 7f19d0865640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f19b006b810 0x7f19b006dcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f19c0036000 tx=0x7f19c003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:47.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:47.975+0000 7f19c97fa640 1 -- 172.21.15.161:0/4277450286 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f19b80540f0 con 0x7f19cc0d0380 2024-03-23T14:35:48.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.145+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f19cc04f4e0 con 0x7f19cc0d0380 2024-03-23T14:35:48.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.146+0000 7f19c97fa640 1 -- 172.21.15.161:0/4277450286 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f19b80577b0 con 0x7f19cc0d0380 2024-03-23T14:35:48.142 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:48.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.147+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f19b006b810 msgr2=0x7f19b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f19b006b810 0x7f19b006dcd0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f19c0036000 tx=0x7f19c003c040 comp rx=0 tx=0).stop 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc0d0380 msgr2=0x7f19cc1445a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc0d0380 0x7f19cc1445a0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f19b800c4a0 tx=0x7f19b800c960 comp rx=0 tx=0).stop 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 shutdown_connections 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f19b006b810 0x7f19b006dcd0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f19cc1552f0 0x7f19cc0cd660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f19cc0d0380 0x7f19cc1445a0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 --2- 172.21.15.161:0/4277450286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f19cc0b2aa0 0x7f19cc15cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 >> 172.21.15.161:0/4277450286 conn(0x7f19cc06b260 msgr2=0x7f19cc15bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 shutdown_connections 2024-03-23T14:35:48.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.148+0000 7f19d2ac9640 1 -- 172.21.15.161:0/4277450286 wait complete. 2024-03-23T14:35:48.154 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:35:48.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.261+0000 7f3a5ad19640 1 Processor -- start 2024-03-23T14:35:48.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.261+0000 7f3a5ad19640 1 -- start start 2024-03-23T14:35:48.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a5ad19640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3a5406eb30 0x7f3a540b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a5ad19640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 0x7f3a5415b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a5ad19640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a54070b90 con 0x7f3a540b3610 2024-03-23T14:35:48.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a5ad19640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3a5415ab90 con 0x7f3a5406eb30 2024-03-23T14:35:48.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a5ad19640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3a540713a0 con 0x7f3a540b4260 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a58ab5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3a5406eb30 0x7f3a540b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a592b6640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3a540b3610 0x7f3a5406d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32844/0 (socket says 172.21.15.161:32844) 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a58ab5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3a5406eb30 0x7f3a540b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41714/0 (socket says 172.21.15.161:41714) 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a592b6640 1 -- 172.21.15.161:0/909430639 learned_addr learned my addr 172.21.15.161:0/909430639 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a53fff640 1 --2- 172.21.15.161:0/909430639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 0x7f3a5415b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a58ab5640 1 -- 172.21.15.161:0/909430639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 msgr2=0x7f3a5415b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a58ab5640 1 --2- 172.21.15.161:0/909430639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 0x7f3a5415b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a58ab5640 1 -- 172.21.15.161:0/909430639 >> v1:172.21.15.161:6789/0 conn(0x7f3a540b3610 legacy=0x7f3a5406d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a58ab5640 1 -- 172.21.15.161:0/909430639 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5415b0d0 con 0x7f3a5406eb30 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.262+0000 7f3a537fe640 1 -- 172.21.15.161:0/909430639 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2053265653 0 0) 0x7f3a54070b90 con 0x7f3a540b3610 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.263+0000 7f3a58ab5640 1 --2- 172.21.15.161:0/909430639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540b4870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3a480097a0 tx=0x7f3a4802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c6dd085099b078 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.263+0000 7f3a537fe640 1 -- 172.21.15.161:0/909430639 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48031830 con 0x7f3a5406eb30 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.263+0000 7f3a537fe640 1 -- 172.21.15.161:0/909430639 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a48031990 con 0x7f3a5406eb30 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.263+0000 7f3a537fe640 1 -- 172.21.15.161:0/909430639 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a48031d20 con 0x7f3a5406eb30 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.264+0000 7f3a5ad19640 1 -- 172.21.15.161:0/909430639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 msgr2=0x7f3a540b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.264+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/909430639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540b4870 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f3a480097a0 tx=0x7f3a4802b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.264+0000 7f3a5ad19640 1 -- 172.21.15.161:0/909430639 shutdown_connections 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.264+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/909430639 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 0x7f3a5415b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.264+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/909430639 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540b4870 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.264+0000 7f3a5ad19640 1 -- 172.21.15.161:0/909430639 >> 172.21.15.161:0/909430639 conn(0x7f3a540b17f0 msgr2=0x7f3a540b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:48.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.264+0000 7f3a5ad19640 1 -- 172.21.15.161:0/909430639 shutdown_connections 2024-03-23T14:35:48.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.264+0000 7f3a5ad19640 1 -- 172.21.15.161:0/909430639 wait complete. 2024-03-23T14:35:48.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.265+0000 7f3a5ad19640 1 Processor -- start 2024-03-23T14:35:48.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.265+0000 7f3a5ad19640 1 -- start start 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a5ad19640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540ce910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a5ad19640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a540b3610 0x7f3a540ccec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a5ad19640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 0x7f3a54159330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a5ad19640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a540cef80 con 0x7f3a5406eb30 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a5ad19640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3a540cf260 con 0x7f3a540b3610 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a5ad19640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3a540cf540 con 0x7f3a540b4260 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540ce910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540ce910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41718/0 (socket says 172.21.15.161:41718) 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 -- 172.21.15.161:0/1516966767 learned_addr learned my addr 172.21.15.161:0/1516966767 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:48.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a53fff640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a540b3610 0x7f3a540ccec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a592b6640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 0x7f3a54159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 -- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 msgr2=0x7f3a54159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 0x7f3a54159330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 -- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a540b3610 msgr2=0x7f3a540ccec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a540b3610 0x7f3a540ccec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 -- 172.21.15.161:0/1516966767 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a48009400 con 0x7f3a5406eb30 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a58ab5640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540ce910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3a480098c0 tx=0x7f3a480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a517fa640 1 -- 172.21.15.161:0/1516966767 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4803a6f0 con 0x7f3a5406eb30 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a517fa640 1 -- 172.21.15.161:0/1516966767 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a4803a870 con 0x7f3a5406eb30 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a54147c20 con 0x7f3a5406eb30 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a517fa640 1 -- 172.21.15.161:0/1516966767 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a4803ac20 con 0x7f3a5406eb30 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.266+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a540cd400 con 0x7f3a5406eb30 2024-03-23T14:35:48.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.268+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a54148140 con 0x7f3a5406eb30 2024-03-23T14:35:48.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.268+0000 7f3a517fa640 1 -- 172.21.15.161:0/1516966767 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3a48030380 con 0x7f3a5406eb30 2024-03-23T14:35:48.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.268+0000 7f3a517fa640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a3806b810 0x7f3a3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.268+0000 7f3a517fa640 1 -- 172.21.15.161:0/1516966767 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f3a480abb00 con 0x7f3a5406eb30 2024-03-23T14:35:48.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.268+0000 7f3a53fff640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a3806b810 0x7f3a3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.269+0000 7f3a53fff640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a3806b810 0x7f3a3806dcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3a541f7910 tx=0x7f3a4000a2a0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.273+0000 7f3a517fa640 1 -- 172.21.15.161:0/1516966767 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3a54148140 con 0x7f3a5406eb30 2024-03-23T14:35:48.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.459+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f3a54148140 con 0x7f3a5406eb30 2024-03-23T14:35:48.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.460+0000 7f3a517fa640 1 -- 172.21.15.161:0/1516966767 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f3a54148140 con 0x7f3a5406eb30 2024-03-23T14:35:48.456 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:35:48.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a3806b810 msgr2=0x7f3a3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a3806b810 0x7f3a3806dcd0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3a541f7910 tx=0x7f3a4000a2a0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 msgr2=0x7f3a540ce910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540ce910 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f3a480098c0 tx=0x7f3a480046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 shutdown_connections 2024-03-23T14:35:48.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a3806b810 0x7f3a3806dcd0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a540b4260 0x7f3a54159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a540b3610 0x7f3a540ccec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 --2- 172.21.15.161:0/1516966767 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a5406eb30 0x7f3a540ce910 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 >> 172.21.15.161:0/1516966767 conn(0x7f3a540b17f0 msgr2=0x7f3a540b0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:48.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 shutdown_connections 2024-03-23T14:35:48.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.462+0000 7f3a5ad19640 1 -- 172.21.15.161:0/1516966767 wait complete. 2024-03-23T14:35:48.469 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:35:48.469 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:48.484 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:48.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.630+0000 7f2086ac2640 1 Processor -- start 2024-03-23T14:35:48.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.630+0000 7f2086ac2640 1 -- start start 2024-03-23T14:35:48.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.630+0000 7f2086ac2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2080145080 0x7f2080072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.630+0000 7f2086ac2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f20801601f0 0x7f208015c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.630+0000 7f2086ac2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20800cf4f0 con 0x7f20801607d0 2024-03-23T14:35:48.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.630+0000 7f2086ac2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2080071010 con 0x7f2080145080 2024-03-23T14:35:48.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.630+0000 7f2086ac2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f20800cef30 con 0x7f20801601f0 2024-03-23T14:35:48.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208485e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2080145080 0x7f2080072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208485e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2080145080 0x7f2080072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41726/0 (socket says 172.21.15.161:41726) 2024-03-23T14:35:48.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208505f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f20801601f0 0x7f208015c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208485e640 1 -- 172.21.15.161:0/2681755074 learned_addr learned my addr 172.21.15.161:0/2681755074 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:48.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208485e640 1 -- 172.21.15.161:0/2681755074 >> v2:172.21.15.183:3300/0 conn(0x7f20801601f0 msgr2=0x7f208015c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:48.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208485e640 1 --2- 172.21.15.161:0/2681755074 >> v2:172.21.15.183:3300/0 conn(0x7f20801601f0 0x7f208015c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208485e640 1 -- 172.21.15.161:0/2681755074 >> v1:172.21.15.183:6789/0 conn(0x7f20801607d0 legacy=0x7f208015fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208485e640 1 -- 172.21.15.161:0/2681755074 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20800cf210 con 0x7f2080145080 2024-03-23T14:35:48.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.631+0000 7f208505f640 1 --2- 172.21.15.161:0/2681755074 >> v2:172.21.15.183:3300/0 conn(0x7f20801601f0 0x7f208015c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:48.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f208485e640 1 --2- 172.21.15.161:0/2681755074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2080145080 0x7f2080072a00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2080070950 tx=0x7f207402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=543cdc41fe4ece15 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f207f7fe640 1 -- 172.21.15.161:0/2681755074 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074031870 con 0x7f2080145080 2024-03-23T14:35:48.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f207f7fe640 1 -- 172.21.15.161:0/2681755074 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20740319d0 con 0x7f2080145080 2024-03-23T14:35:48.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f207f7fe640 1 -- 172.21.15.161:0/2681755074 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074031d60 con 0x7f2080145080 2024-03-23T14:35:48.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f2086ac2640 1 -- 172.21.15.161:0/2681755074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2080145080 msgr2=0x7f2080072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f2086ac2640 1 --2- 172.21.15.161:0/2681755074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2080145080 0x7f2080072a00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f2080070950 tx=0x7f207402b910 comp rx=0 tx=0).stop 2024-03-23T14:35:48.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f2086ac2640 1 -- 172.21.15.161:0/2681755074 shutdown_connections 2024-03-23T14:35:48.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f2086ac2640 1 --2- 172.21.15.161:0/2681755074 >> v2:172.21.15.183:3300/0 conn(0x7f20801601f0 0x7f208015c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f2086ac2640 1 --2- 172.21.15.161:0/2681755074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2080145080 0x7f2080072a00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f2086ac2640 1 -- 172.21.15.161:0/2681755074 >> 172.21.15.161:0/2681755074 conn(0x7f20800b17f0 msgr2=0x7f20800b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:48.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f2086ac2640 1 -- 172.21.15.161:0/2681755074 shutdown_connections 2024-03-23T14:35:48.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.632+0000 7f2086ac2640 1 -- 172.21.15.161:0/2681755074 wait complete. 2024-03-23T14:35:48.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.634+0000 7f2086ac2640 1 Processor -- start 2024-03-23T14:35:48.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.634+0000 7f2086ac2640 1 -- start start 2024-03-23T14:35:48.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.634+0000 7f2086ac2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2080145080 0x7f208013f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.634+0000 7f2086ac2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20801601f0 0x7f20800aed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.634+0000 7f2086ac2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20801607d0 0x7f208013bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.634+0000 7f2086ac2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f208013cd00 con 0x7f20801607d0 2024-03-23T14:35:48.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.634+0000 7f2086ac2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f208013ca20 con 0x7f20801601f0 2024-03-23T14:35:48.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f2086ac2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f208013e600 con 0x7f2080145080 2024-03-23T14:35:48.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208505f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20801607d0 0x7f208013bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2080145080 0x7f208013f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2080145080 0x7f208013f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:52184/0 (socket says 172.21.15.161:52184) 2024-03-23T14:35:48.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208505f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20801607d0 0x7f208013bfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41736/0 (socket says 172.21.15.161:41736) 2024-03-23T14:35:48.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 -- 172.21.15.161:0/342439169 learned_addr learned my addr 172.21.15.161:0/342439169 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:48.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f207ffff640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20801601f0 0x7f20800aed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 -- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20801601f0 msgr2=0x7f20800aed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20801601f0 0x7f20800aed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 -- 172.21.15.161:0/342439169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20801607d0 msgr2=0x7f208013bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20801607d0 0x7f208013bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 -- 172.21.15.161:0/342439169 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2078009a70 con 0x7f2080145080 2024-03-23T14:35:48.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208505f640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20801607d0 0x7f208013bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:48.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f208485e640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2080145080 0x7f208013f0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f207402b950 tx=0x7f20740046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f207d7fa640 1 -- 172.21.15.161:0/342439169 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207403abe0 con 0x7f2080145080 2024-03-23T14:35:48.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f207d7fa640 1 -- 172.21.15.161:0/342439169 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f207403ad60 con 0x7f2080145080 2024-03-23T14:35:48.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20740094d0 con 0x7f2080145080 2024-03-23T14:35:48.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f207d7fa640 1 -- 172.21.15.161:0/342439169 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2074044630 con 0x7f2080145080 2024-03-23T14:35:48.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.635+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f208013ebc0 con 0x7f2080145080 2024-03-23T14:35:48.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.636+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f208006cb70 con 0x7f2080145080 2024-03-23T14:35:48.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.637+0000 7f207d7fa640 1 -- 172.21.15.161:0/342439169 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f207404d450 con 0x7f2080145080 2024-03-23T14:35:48.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.637+0000 7f207d7fa640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f206406b810 0x7f206406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.637+0000 7f207d7fa640 1 -- 172.21.15.161:0/342439169 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f207402d040 con 0x7f2080145080 2024-03-23T14:35:48.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.637+0000 7f207ffff640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f206406b810 0x7f206406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.638+0000 7f207ffff640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f206406b810 0x7f206406dcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f20800b0b10 tx=0x7f206c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.646 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.642+0000 7f207d7fa640 1 -- 172.21.15.161:0/342439169 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f20740770f0 con 0x7f2080145080 2024-03-23T14:35:48.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.814+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f20800735e0 con 0x7f2080145080 2024-03-23T14:35:48.811 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.815+0000 7f207d7fa640 1 -- 172.21.15.161:0/342439169 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f207407a7b0 con 0x7f2080145080 2024-03-23T14:35:48.812 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:48.812 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:48.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f206406b810 msgr2=0x7f206406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f206406b810 0x7f206406dcd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f20800b0b10 tx=0x7f206c009040 comp rx=0 tx=0).stop 2024-03-23T14:35:48.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2080145080 msgr2=0x7f208013f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2080145080 0x7f208013f0e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f207402b950 tx=0x7f20740046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 shutdown_connections 2024-03-23T14:35:48.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f206406b810 0x7f206406dcd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20801607d0 0x7f208013bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20801601f0 0x7f20800aed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 --2- 172.21.15.161:0/342439169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2080145080 0x7f208013f0e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 >> 172.21.15.161:0/342439169 conn(0x7f20800b17f0 msgr2=0x7f208006cfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:48.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 shutdown_connections 2024-03-23T14:35:48.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.818+0000 7f2086ac2640 1 -- 172.21.15.161:0/342439169 wait complete. 2024-03-23T14:35:48.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:48.827 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:48.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.935+0000 7fd287a64640 1 Processor -- start 2024-03-23T14:35:48.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.935+0000 7fd287a64640 1 -- start start 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd287a64640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28015a2b0 0x7fd28015df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd287a64640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd280071370 0x7fd280071800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd287a64640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd280070ee0 con 0x7fd28006cd20 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd287a64640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fd2800b33b0 con 0x7fd280071370 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd287a64640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd2800b3650 con 0x7fd28015a2b0 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd284fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd28006cd20 0x7fd28015e4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32850/0 (socket says 172.21.15.161:32850) 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd284fff640 1 -- 172.21.15.161:0/2934998715 learned_addr learned my addr 172.21.15.161:0/2934998715 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd285800640 1 --2- 172.21.15.161:0/2934998715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28015a2b0 0x7fd28015df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd286001640 1 --2- 172.21.15.161:0/2934998715 >> v2:172.21.15.183:3300/0 conn(0x7fd280071370 0x7fd280071800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd277fff640 1 -- 172.21.15.161:0/2934998715 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062444396 0 0) 0x7fd280070ee0 con 0x7fd28006cd20 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd277fff640 1 -- 172.21.15.161:0/2934998715 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd26c003520 con 0x7fd28006cd20 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd285800640 1 -- 172.21.15.161:0/2934998715 >> v2:172.21.15.183:3300/0 conn(0x7fd280071370 msgr2=0x7fd280071800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd285800640 1 --2- 172.21.15.161:0/2934998715 >> v2:172.21.15.183:3300/0 conn(0x7fd280071370 0x7fd280071800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd285800640 1 -- 172.21.15.161:0/2934998715 >> v1:172.21.15.161:6789/0 conn(0x7fd28006cd20 legacy=0x7fd28015e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.936+0000 7fd285800640 1 -- 172.21.15.161:0/2934998715 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2800b3930 con 0x7fd28015a2b0 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.937+0000 7fd285800640 1 --2- 172.21.15.161:0/2934998715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28015a2b0 0x7fd28015df80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0097a0 tx=0x7fd27c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ee035fea87280a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.937+0000 7fd277fff640 1 -- 172.21.15.161:0/2934998715 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c031830 con 0x7fd28015a2b0 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.937+0000 7fd277fff640 1 -- 172.21.15.161:0/2934998715 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27c031990 con 0x7fd28015a2b0 2024-03-23T14:35:48.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.937+0000 7fd277fff640 1 -- 172.21.15.161:0/2934998715 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27c031d20 con 0x7fd28015a2b0 2024-03-23T14:35:48.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.938+0000 7fd287a64640 1 -- 172.21.15.161:0/2934998715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28015a2b0 msgr2=0x7fd28015df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.938+0000 7fd287a64640 1 --2- 172.21.15.161:0/2934998715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28015a2b0 0x7fd28015df80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd27c0097a0 tx=0x7fd27c02b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.938+0000 7fd287a64640 1 -- 172.21.15.161:0/2934998715 shutdown_connections 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.938+0000 7fd287a64640 1 --2- 172.21.15.161:0/2934998715 >> v2:172.21.15.183:3300/0 conn(0x7fd280071370 0x7fd280071800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.938+0000 7fd287a64640 1 --2- 172.21.15.161:0/2934998715 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28015a2b0 0x7fd28015df80 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.938+0000 7fd287a64640 1 -- 172.21.15.161:0/2934998715 >> 172.21.15.161:0/2934998715 conn(0x7fd28006b260 msgr2=0x7fd28006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.938+0000 7fd287a64640 1 -- 172.21.15.161:0/2934998715 shutdown_connections 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.938+0000 7fd287a64640 1 -- 172.21.15.161:0/2934998715 wait complete. 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.939+0000 7fd287a64640 1 Processor -- start 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.939+0000 7fd287a64640 1 -- start start 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd287a64640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28006cd20 0x7fd280146aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd287a64640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280071370 0x7fd2800ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd287a64640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015a2b0 0x7fd2801660f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd287a64640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2800ab060 con 0x7fd28015a2b0 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd287a64640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd280146fe0 con 0x7fd280071370 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd287a64640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd280166b90 con 0x7fd28006cd20 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015a2b0 0x7fd2801660f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015a2b0 0x7fd2801660f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41740/0 (socket says 172.21.15.161:41740) 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 -- 172.21.15.161:0/859737018 learned_addr learned my addr 172.21.15.161:0/859737018 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd284fff640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280071370 0x7fd2800ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd285800640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28006cd20 0x7fd280146aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 -- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28006cd20 msgr2=0x7fd280146aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28006cd20 0x7fd280146aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 -- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280071370 msgr2=0x7fd2800ab4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:48.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280071370 0x7fd2800ab4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:48.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 -- 172.21.15.161:0/859737018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd27c009400 con 0x7fd28015a2b0 2024-03-23T14:35:48.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.940+0000 7fd286001640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015a2b0 0x7fd2801660f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd278009dd0 tx=0x7fd27800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.941+0000 7fd275ffb640 1 -- 172.21.15.161:0/859737018 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278014480 con 0x7fd28015a2b0 2024-03-23T14:35:48.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.941+0000 7fd275ffb640 1 -- 172.21.15.161:0/859737018 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27800f040 con 0x7fd28015a2b0 2024-03-23T14:35:48.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.941+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2800aaa10 con 0x7fd28015a2b0 2024-03-23T14:35:48.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.941+0000 7fd275ffb640 1 -- 172.21.15.161:0/859737018 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd278014840 con 0x7fd28015a2b0 2024-03-23T14:35:48.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.941+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd280166e70 con 0x7fd28015a2b0 2024-03-23T14:35:48.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.942+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2800ab210 con 0x7fd28015a2b0 2024-03-23T14:35:48.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.942+0000 7fd275ffb640 1 -- 172.21.15.161:0/859737018 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd2780149a0 con 0x7fd28015a2b0 2024-03-23T14:35:48.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.943+0000 7fd275ffb640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26406b810 0x7fd26406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:48.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.943+0000 7fd275ffb640 1 -- 172.21.15.161:0/859737018 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fd27808ac80 con 0x7fd28015a2b0 2024-03-23T14:35:48.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.943+0000 7fd285800640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26406b810 0x7fd26406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:48.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.943+0000 7fd285800640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26406b810 0x7fd26406dcd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd27c036000 tx=0x7fd27c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:48.942 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:48.947+0000 7fd275ffb640 1 -- 172.21.15.161:0/859737018 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd278055760 con 0x7fd28015a2b0 2024-03-23T14:35:49.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.116+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd28006d4c0 con 0x7fd28015a2b0 2024-03-23T14:35:49.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.117+0000 7fd275ffb640 1 -- 172.21.15.161:0/859737018 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fd278058e20 con 0x7fd28015a2b0 2024-03-23T14:35:49.113 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26406b810 msgr2=0x7fd26406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26406b810 0x7fd26406dcd0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd27c036000 tx=0x7fd27c03c040 comp rx=0 tx=0).stop 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015a2b0 msgr2=0x7fd2801660f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015a2b0 0x7fd2801660f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fd278009dd0 tx=0x7fd27800c490 comp rx=0 tx=0).stop 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 shutdown_connections 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26406b810 0x7fd26406dcd0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015a2b0 0x7fd2801660f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280071370 0x7fd2800ab4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 --2- 172.21.15.161:0/859737018 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd28006cd20 0x7fd280146aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 >> 172.21.15.161:0/859737018 conn(0x7fd28006b260 msgr2=0x7fd2800b1340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 shutdown_connections 2024-03-23T14:35:49.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.119+0000 7fd287a64640 1 -- 172.21.15.161:0/859737018 wait complete. 2024-03-23T14:35:49.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.234+0000 7f2129855640 1 Processor -- start 2024-03-23T14:35:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.234+0000 7f2129855640 1 -- start start 2024-03-23T14:35:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f2129855640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21240b5530 0x7f21240b2c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f2129855640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21240d4600 con 0x7f21240cfaa0 2024-03-23T14:35:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f2129855640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21240cff80 con 0x7f21240d08e0 2024-03-23T14:35:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f2129855640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f21240d4850 con 0x7f21240b5530 2024-03-23T14:35:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f21237fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21240b5530 0x7f21240b2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f21237fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21240b5530 0x7f21240b2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41752/0 (socket says 172.21.15.161:41752) 2024-03-23T14:35:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f21227fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f21240cfaa0 0x7f21240b3c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32858/0 (socket says 172.21.15.161:32858) 2024-03-23T14:35:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f21237fe640 1 -- 172.21.15.161:0/62485179 learned_addr learned my addr 172.21.15.161:0/62485179 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f21237fe640 1 -- 172.21.15.161:0/62485179 >> v1:172.21.15.183:6789/0 conn(0x7f21240d08e0 legacy=0x7f21240d0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f21237fe640 1 -- 172.21.15.161:0/62485179 >> v1:172.21.15.161:6789/0 conn(0x7f21240cfaa0 legacy=0x7f21240b3c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f2121ffb640 1 -- 172.21.15.161:0/62485179 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2013739020 0 0) 0x7f21240cff80 con 0x7f21240d08e0 2024-03-23T14:35:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.235+0000 7f21237fe640 1 -- 172.21.15.161:0/62485179 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21240d4b30 con 0x7f21240b5530 2024-03-23T14:35:49.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f21237fe640 1 --2- 172.21.15.161:0/62485179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f21240b2c50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f211c009e80 tx=0x7f211c02bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4dac09cf7cf6f35 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f2121ffb640 1 -- 172.21.15.161:0/62485179 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c02caf0 con 0x7f21240b5530 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f2121ffb640 1 -- 172.21.15.161:0/62485179 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f211c02cc50 con 0x7f21240b5530 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f2121ffb640 1 -- 172.21.15.161:0/62485179 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f211c0343f0 con 0x7f21240b5530 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f2129855640 1 -- 172.21.15.161:0/62485179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 msgr2=0x7f21240b2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f2129855640 1 --2- 172.21.15.161:0/62485179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f21240b2c50 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f211c009e80 tx=0x7f211c02bf50 comp rx=0 tx=0).stop 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f2129855640 1 -- 172.21.15.161:0/62485179 shutdown_connections 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f2129855640 1 --2- 172.21.15.161:0/62485179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f21240b2c50 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.236+0000 7f2129855640 1 -- 172.21.15.161:0/62485179 >> 172.21.15.161:0/62485179 conn(0x7f212409d220 msgr2=0x7f21240d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.237+0000 7f2129855640 1 -- 172.21.15.161:0/62485179 shutdown_connections 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.237+0000 7f2129855640 1 -- 172.21.15.161:0/62485179 wait complete. 2024-03-23T14:35:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.238+0000 7f2129855640 1 Processor -- start 2024-03-23T14:35:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.238+0000 7f2129855640 1 -- start start 2024-03-23T14:35:49.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.238+0000 7f2129855640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f2124153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.238+0000 7f2129855640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21240cfaa0 0x7f21241f8000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.238+0000 7f2129855640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21240d08e0 0x7f21240b6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.238+0000 7f2129855640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f21241539c0 con 0x7f21240b5530 2024-03-23T14:35:49.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.238+0000 7f2129855640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f212415c800 con 0x7f21240cfaa0 2024-03-23T14:35:49.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.238+0000 7f2129855640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f21240b9810 con 0x7f21240d08e0 2024-03-23T14:35:49.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f2124153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f2124153480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41756/0 (socket says 172.21.15.161:41756) 2024-03-23T14:35:49.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 -- 172.21.15.161:0/118334949 learned_addr learned my addr 172.21.15.161:0/118334949 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:49.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f21227fc640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21240cfaa0 0x7f21241f8000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f21237fe640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21240d08e0 0x7f21240b6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 -- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21240d08e0 msgr2=0x7f21240b6ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21240d08e0 0x7f21240b6ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 -- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21240cfaa0 msgr2=0x7f21241f8000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21240cfaa0 0x7f21241f8000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 -- 172.21.15.161:0/118334949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f211c009b30 con 0x7f21240b5530 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2122ffd640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f2124153480 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f211400c410 tx=0x7f211400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f210bfff640 1 -- 172.21.15.161:0/118334949 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f21140035f0 con 0x7f21240b5530 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f210bfff640 1 -- 172.21.15.161:0/118334949 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2114003770 con 0x7f21240b5530 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21241f8540 con 0x7f21240b5530 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f210bfff640 1 -- 172.21.15.161:0/118334949 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2114003b20 con 0x7f21240b5530 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.239+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21240cb160 con 0x7f21240b5530 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.240+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21240cafa0 con 0x7f21240b5530 2024-03-23T14:35:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.241+0000 7f210bfff640 1 -- 172.21.15.161:0/118334949 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2114003c80 con 0x7f21240b5530 2024-03-23T14:35:49.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.241+0000 7f210bfff640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f211006b810 0x7f211006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.241+0000 7f210bfff640 1 -- 172.21.15.161:0/118334949 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f21140896b0 con 0x7f21240b5530 2024-03-23T14:35:49.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.241+0000 7f21227fc640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f211006b810 0x7f211006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.242+0000 7f21227fc640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f211006b810 0x7f211006dcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f21240d25e0 tx=0x7f2118007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.245+0000 7f210bfff640 1 -- 172.21.15.161:0/118334949 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2114054210 con 0x7f21240b5530 2024-03-23T14:35:49.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.417+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f21240a1f00 con 0x7f21240b5530 2024-03-23T14:35:49.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.417+0000 7f210bfff640 1 -- 172.21.15.161:0/118334949 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f21140578d0 con 0x7f21240b5530 2024-03-23T14:35:49.413 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:49.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f211006b810 msgr2=0x7f211006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f211006b810 0x7f211006dcd0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f21240d25e0 tx=0x7f2118007e80 comp rx=0 tx=0).stop 2024-03-23T14:35:49.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 msgr2=0x7f2124153480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f2124153480 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f211400c410 tx=0x7f211400c8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 shutdown_connections 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f211006b810 0x7f211006dcd0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21240d08e0 0x7f21240b6ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21240cfaa0 0x7f21241f8000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 --2- 172.21.15.161:0/118334949 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21240b5530 0x7f2124153480 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 >> 172.21.15.161:0/118334949 conn(0x7f212409d220 msgr2=0x7f21240b6270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 shutdown_connections 2024-03-23T14:35:49.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.419+0000 7f2129855640 1 -- 172.21.15.161:0/118334949 wait complete. 2024-03-23T14:35:49.426 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:49.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.533+0000 7f725cdf4640 1 Processor -- start 2024-03-23T14:35:49.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.533+0000 7f725cdf4640 1 -- start start 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.534+0000 7f725cdf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72580d1270 0x7f72580d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.534+0000 7f725cdf4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f72580d29d0 0x7f72580b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.534+0000 7f725cdf4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7258161c00 con 0x7f7258161820 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.534+0000 7f725cdf4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f72580d2e20 con 0x7f72580d29d0 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.534+0000 7f725cdf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f72580cea30 con 0x7f72580d1270 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.534+0000 7f725659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7258161820 0x7f72580d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32868/0 (socket says 172.21.15.161:32868) 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.534+0000 7f725659c640 1 -- 172.21.15.161:0/678311701 learned_addr learned my addr 172.21.15.161:0/678311701 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.535+0000 7f7255d9b640 1 --2- 172.21.15.161:0/678311701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72580d1270 0x7f72580d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.535+0000 7f7256d9d640 1 --2- 172.21.15.161:0/678311701 >> v2:172.21.15.183:3300/0 conn(0x7f72580d29d0 0x7f72580b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.535+0000 7f725559a640 1 -- 172.21.15.161:0/678311701 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3176003861 0 0) 0x7f7258161c00 con 0x7f7258161820 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.535+0000 7f725559a640 1 -- 172.21.15.161:0/678311701 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7240003520 con 0x7f7258161820 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.535+0000 7f7255d9b640 1 -- 172.21.15.161:0/678311701 >> v2:172.21.15.183:3300/0 conn(0x7f72580d29d0 msgr2=0x7f72580b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:49.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.535+0000 7f7255d9b640 1 --2- 172.21.15.161:0/678311701 >> v2:172.21.15.183:3300/0 conn(0x7f72580d29d0 0x7f72580b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.535+0000 7f7255d9b640 1 -- 172.21.15.161:0/678311701 >> v1:172.21.15.161:6789/0 conn(0x7f7258161820 legacy=0x7f72580d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.535+0000 7f7255d9b640 1 -- 172.21.15.161:0/678311701 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72580cfd70 con 0x7f72580d1270 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f7255d9b640 1 --2- 172.21.15.161:0/678311701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72580d1270 0x7f72580d2490 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f724c0097a0 tx=0x7f724c02b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f7daa4b48793476 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725559a640 1 -- 172.21.15.161:0/678311701 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c031830 con 0x7f72580d1270 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725559a640 1 -- 172.21.15.161:0/678311701 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c031990 con 0x7f72580d1270 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725559a640 1 -- 172.21.15.161:0/678311701 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c031d20 con 0x7f72580d1270 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725cdf4640 1 -- 172.21.15.161:0/678311701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72580d1270 msgr2=0x7f72580d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725cdf4640 1 --2- 172.21.15.161:0/678311701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72580d1270 0x7f72580d2490 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f724c0097a0 tx=0x7f724c02b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725cdf4640 1 -- 172.21.15.161:0/678311701 shutdown_connections 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725cdf4640 1 --2- 172.21.15.161:0/678311701 >> v2:172.21.15.183:3300/0 conn(0x7f72580d29d0 0x7f72580b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725cdf4640 1 --2- 172.21.15.161:0/678311701 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f72580d1270 0x7f72580d2490 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725cdf4640 1 -- 172.21.15.161:0/678311701 >> 172.21.15.161:0/678311701 conn(0x7f725809d190 msgr2=0x7f72580d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:49.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725cdf4640 1 -- 172.21.15.161:0/678311701 shutdown_connections 2024-03-23T14:35:49.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.536+0000 7f725cdf4640 1 -- 172.21.15.161:0/678311701 wait complete. 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.538+0000 7f725cdf4640 1 Processor -- start 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.538+0000 7f725cdf4640 1 -- start start 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.538+0000 7f725cdf4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72580d1270 0x7f72581f51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f725cdf4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72580d29d0 0x7f725815a640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f725cdf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7258161820 0x7f72580c9200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f725cdf4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f72580cb950 con 0x7f72580d29d0 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f725cdf4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f72580cbc30 con 0x7f72580d1270 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f725cdf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f72580cbf10 con 0x7f7258161820 2024-03-23T14:35:49.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72580d29d0 0x7f725815a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72580d29d0 0x7f725815a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41772/0 (socket says 172.21.15.161:41772) 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 -- 172.21.15.161:0/3690336770 learned_addr learned my addr 172.21.15.161:0/3690336770 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f725659c640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72580d1270 0x7f72581f51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7256d9d640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7258161820 0x7f72580c9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 -- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7258161820 msgr2=0x7f72580c9200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7258161820 0x7f72580c9200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 -- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72580d1270 msgr2=0x7f72581f51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72580d1270 0x7f72581f51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 -- 172.21.15.161:0/3690336770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f724c009400 con 0x7f72580d29d0 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7255d9b640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72580d29d0 0x7f725815a640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f724c034040 tx=0x7f724c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7246ffd640 1 -- 172.21.15.161:0/3690336770 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c03a830 con 0x7f72580d29d0 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7246ffd640 1 -- 172.21.15.161:0/3690336770 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f724c03a9b0 con 0x7f72580d29d0 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72580cc1f0 con 0x7f72580d29d0 2024-03-23T14:35:49.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f7246ffd640 1 -- 172.21.15.161:0/3690336770 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f724c03ad60 con 0x7f72580d29d0 2024-03-23T14:35:49.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.539+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72580cb0d0 con 0x7f72580d29d0 2024-03-23T14:35:49.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.540+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72580d27c0 con 0x7f72580d29d0 2024-03-23T14:35:49.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.541+0000 7f7246ffd640 1 -- 172.21.15.161:0/3690336770 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f724c04d440 con 0x7f72580d29d0 2024-03-23T14:35:49.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.541+0000 7f7246ffd640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f723806b840 0x7f723806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.541+0000 7f7246ffd640 1 -- 172.21.15.161:0/3690336770 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f724c0ac0d0 con 0x7f72580d29d0 2024-03-23T14:35:49.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.541+0000 7f725659c640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f723806b840 0x7f723806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.542+0000 7f725659c640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f723806b840 0x7f723806dd00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7248005c00 tx=0x7f7248009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.545+0000 7f7246ffd640 1 -- 172.21.15.161:0/3690336770 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f724c076cf0 con 0x7f72580d29d0 2024-03-23T14:35:49.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.716+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f725804f4e0 con 0x7f72580d29d0 2024-03-23T14:35:49.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.717+0000 7f7246ffd640 1 -- 172.21.15.161:0/3690336770 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f724c07a3b0 con 0x7f72580d29d0 2024-03-23T14:35:49.713 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f723806b840 msgr2=0x7f723806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f723806b840 0x7f723806dd00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7248005c00 tx=0x7f7248009040 comp rx=0 tx=0).stop 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72580d29d0 msgr2=0x7f725815a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72580d29d0 0x7f725815a640 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f724c034040 tx=0x7f724c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 shutdown_connections 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f723806b840 0x7f723806dd00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7258161820 0x7f72580c9200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f72580d29d0 0x7f725815a640 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 --2- 172.21.15.161:0/3690336770 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f72580d1270 0x7f72581f51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 >> 172.21.15.161:0/3690336770 conn(0x7f725809d190 msgr2=0x7f72580d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:49.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 shutdown_connections 2024-03-23T14:35:49.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.719+0000 7f725cdf4640 1 -- 172.21.15.161:0/3690336770 wait complete. 2024-03-23T14:35:49.726 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:35:49.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.831+0000 7f1088eae640 1 Processor -- start 2024-03-23T14:35:49.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.831+0000 7f1088eae640 1 -- start start 2024-03-23T14:35:49.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.832+0000 7f1088eae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 0x7f10840cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.832+0000 7f1088eae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f10840cf9d0 0x7f10840d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.832+0000 7f1088eae640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10840d0460 con 0x7f10840d1470 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.832+0000 7f1088eae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f10840d1050 con 0x7f10840cf9d0 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.832+0000 7f1088eae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f10840d06b0 con 0x7f108415e400 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.832+0000 7f1081d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f10840cf9d0 0x7f10840d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.833+0000 7f1081d9b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f10840cf9d0 0x7f10840d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41788/0 (socket says 172.21.15.161:41788) 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.833+0000 7f1081d9b640 1 -- 172.21.15.161:0/2430246705 learned_addr learned my addr 172.21.15.161:0/2430246705 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.833+0000 7f108259c640 1 --2- 172.21.15.161:0/2430246705 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 0x7f10840cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.833+0000 7f1081d9b640 1 -- 172.21.15.161:0/2430246705 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 msgr2=0x7f10840cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.833+0000 7f1081d9b640 1 --2- 172.21.15.161:0/2430246705 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 0x7f10840cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.833+0000 7f1081d9b640 1 -- 172.21.15.161:0/2430246705 >> v1:172.21.15.183:6789/0 conn(0x7f10840d1470 legacy=0x7f10840b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.833+0000 7f1081d9b640 1 -- 172.21.15.161:0/2430246705 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10840d0990 con 0x7f10840cf9d0 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.833+0000 7f1081d9b640 1 --2- 172.21.15.161:0/2430246705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840d2430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1078009870 tx=0x7f107802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68a45a40d9b2cbb9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f108159a640 1 -- 172.21.15.161:0/2430246705 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078030840 con 0x7f10840cf9d0 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f108159a640 1 -- 172.21.15.161:0/2430246705 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10780309a0 con 0x7f10840cf9d0 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f108159a640 1 -- 172.21.15.161:0/2430246705 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1078030d30 con 0x7f10840cf9d0 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f1088eae640 1 -- 172.21.15.161:0/2430246705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 msgr2=0x7f10840d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f1088eae640 1 --2- 172.21.15.161:0/2430246705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840d2430 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1078009870 tx=0x7f107802b980 comp rx=0 tx=0).stop 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f1088eae640 1 -- 172.21.15.161:0/2430246705 shutdown_connections 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f1088eae640 1 --2- 172.21.15.161:0/2430246705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840d2430 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f1088eae640 1 --2- 172.21.15.161:0/2430246705 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 0x7f10840cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f1088eae640 1 -- 172.21.15.161:0/2430246705 >> 172.21.15.161:0/2430246705 conn(0x7f108409d190 msgr2=0x7f10840d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f1088eae640 1 -- 172.21.15.161:0/2430246705 shutdown_connections 2024-03-23T14:35:49.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.834+0000 7f1088eae640 1 -- 172.21.15.161:0/2430246705 wait complete. 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1088eae640 1 Processor -- start 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1088eae640 1 -- start start 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1088eae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840c85c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1088eae640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10840d1470 0x7f10841591f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1088eae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 0x7f108415fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1088eae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f10840c8ea0 con 0x7f10840cf9d0 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1088eae640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f10840c9180 con 0x7f10840d1470 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1088eae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f10840b4720 con 0x7f108415e400 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f108259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840c85c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f108259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840c85c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41804/0 (socket says 172.21.15.161:41804) 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f108259c640 1 -- 172.21.15.161:0/3173291497 learned_addr learned my addr 172.21.15.161:0/3173291497 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:49.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1082d9d640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 0x7f108415fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.836+0000 7f1081d9b640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10840d1470 0x7f10841591f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f108259c640 1 -- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 msgr2=0x7f108415fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f108259c640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 0x7f108415fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f108259c640 1 -- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10840d1470 msgr2=0x7f10841591f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f108259c640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10840d1470 0x7f10841591f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f108259c640 1 -- 172.21.15.161:0/3173291497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10780094d0 con 0x7f10840cf9d0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f108259c640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840c85c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f107400efc0 tx=0x7f107400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f1072ffd640 1 -- 172.21.15.161:0/3173291497 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1074009b80 con 0x7f10840cf9d0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f1072ffd640 1 -- 172.21.15.161:0/3173291497 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107400f040 con 0x7f10840cf9d0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10840b4a00 con 0x7f10840cf9d0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f1072ffd640 1 -- 172.21.15.161:0/3173291497 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10740146a0 con 0x7f10840cf9d0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.837+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10840c8b00 con 0x7f10840cf9d0 2024-03-23T14:35:49.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.838+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10840a1e00 con 0x7f10840cf9d0 2024-03-23T14:35:49.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.838+0000 7f1072ffd640 1 -- 172.21.15.161:0/3173291497 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1074014800 con 0x7f10840cf9d0 2024-03-23T14:35:49.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.839+0000 7f1072ffd640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f106406b840 0x7f106406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:49.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.839+0000 7f1072ffd640 1 -- 172.21.15.161:0/3173291497 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f107408ab90 con 0x7f10840cf9d0 2024-03-23T14:35:49.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.839+0000 7f1081d9b640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f106406b840 0x7f106406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:49.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.839+0000 7f1081d9b640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f106406b840 0x7f106406dd00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1078036000 tx=0x7f1078008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:49.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:49.843+0000 7f1072ffd640 1 -- 172.21.15.161:0/3173291497 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1074055660 con 0x7f10840cf9d0 2024-03-23T14:35:50.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.025+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f108404f550 con 0x7f10840cf9d0 2024-03-23T14:35:50.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.025+0000 7f1072ffd640 1 -- 172.21.15.161:0/3173291497 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7f1074058d20 con 0x7f10840cf9d0 2024-03-23T14:35:50.021 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:35:50.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f106406b840 msgr2=0x7f106406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f106406b840 0x7f106406dd00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f1078036000 tx=0x7f1078008030 comp rx=0 tx=0).stop 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 msgr2=0x7f10840c85c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840c85c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f107400efc0 tx=0x7f107400c490 comp rx=0 tx=0).stop 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 shutdown_connections 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f106406b840 0x7f106406dd00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f108415e400 0x7f108415fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f10840d1470 0x7f10841591f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 --2- 172.21.15.161:0/3173291497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f10840cf9d0 0x7f10840c85c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.027+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 >> 172.21.15.161:0/3173291497 conn(0x7f108409d190 msgr2=0x7f10840b1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:50.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.028+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 shutdown_connections 2024-03-23T14:35:50.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.028+0000 7f1088eae640 1 -- 172.21.15.161:0/3173291497 wait complete. 2024-03-23T14:35:50.034 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-23T14:35:50.034 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:50.035 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.2 2024-03-23T14:35:50.062 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:35:50.062 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:35:50.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:50.062 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T14:35:50.118 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:35:50.118 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:50.134 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:35:50.134 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:50.190 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:50.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.294+0000 7f211869d640 1 Processor -- start 2024-03-23T14:35:50.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.294+0000 7f211869d640 1 -- start start 2024-03-23T14:35:50.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f211869d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21100d08e0 0x7f21100d0d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f211869d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21100ce4f0 0x7f21100ce960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f211869d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f21100ceea0 con 0x7f211015e240 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f211869d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f211006f2d0 con 0x7f21100ce4f0 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f211869d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f21100cf0d0 con 0x7f21100d08e0 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f2116c3a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21100ce4f0 0x7f21100ce960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f2116c3a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f21100ce4f0 0x7f21100ce960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41808/0 (socket says 172.21.15.161:41808) 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f2116c3a640 1 -- 172.21.15.161:0/1219983432 learned_addr learned my addr 172.21.15.161:0/1219983432 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f2116439640 1 --1- 172.21.15.161:0/1219983432 >> v1:172.21.15.161:6789/0 conn(0x7f211015e240 0x7f211006c640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:32882/0 (socket says 172.21.15.161:32882) 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.295+0000 7f2115c38640 1 --2- 172.21.15.161:0/1219983432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21100d08e0 0x7f21100d0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.296+0000 7f2116c3a640 1 -- 172.21.15.161:0/1219983432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21100d08e0 msgr2=0x7f21100d0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.296+0000 7f2116c3a640 1 --2- 172.21.15.161:0/1219983432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21100d08e0 0x7f21100d0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.296+0000 7f2116c3a640 1 -- 172.21.15.161:0/1219983432 >> v1:172.21.15.161:6789/0 conn(0x7f211015e240 legacy=0x7f211006c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.296+0000 7f2115437640 1 -- 172.21.15.161:0/1219983432 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3788021683 0 0) 0x7f21100ceea0 con 0x7f211015e240 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.296+0000 7f2116c3a640 1 -- 172.21.15.161:0/1219983432 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21100cf3b0 con 0x7f21100ce4f0 2024-03-23T14:35:50.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.296+0000 7f2116c3a640 1 --2- 172.21.15.161:0/1219983432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100ce960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2108009870 tx=0x7f210802b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29210f1a4e495900 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f2115437640 1 -- 172.21.15.161:0/1219983432 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108030840 con 0x7f21100ce4f0 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f2115437640 1 -- 172.21.15.161:0/1219983432 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f21080309a0 con 0x7f21100ce4f0 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f2115437640 1 -- 172.21.15.161:0/1219983432 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2108030d30 con 0x7f21100ce4f0 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f211869d640 1 -- 172.21.15.161:0/1219983432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 msgr2=0x7f21100ce960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f211869d640 1 --2- 172.21.15.161:0/1219983432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100ce960 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f2108009870 tx=0x7f210802b980 comp rx=0 tx=0).stop 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f211869d640 1 -- 172.21.15.161:0/1219983432 shutdown_connections 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f211869d640 1 --2- 172.21.15.161:0/1219983432 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100ce960 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f211869d640 1 --2- 172.21.15.161:0/1219983432 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f21100d08e0 0x7f21100d0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f211869d640 1 -- 172.21.15.161:0/1219983432 >> 172.21.15.161:0/1219983432 conn(0x7f211009aa40 msgr2=0x7f211009ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f211869d640 1 -- 172.21.15.161:0/1219983432 shutdown_connections 2024-03-23T14:35:50.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.297+0000 7f211869d640 1 -- 172.21.15.161:0/1219983432 wait complete. 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.298+0000 7f211869d640 1 Processor -- start 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.298+0000 7f211869d640 1 -- start start 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f211869d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f211869d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21100d08e0 0x7f2110153ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f211869d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f211015e240 0x7f21100cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f211869d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f21100b1ab0 con 0x7f21100ce4f0 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f211869d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f21100b1d90 con 0x7f21100d08e0 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f211869d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f21100b2070 con 0x7f211015e240 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f2116439640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f2116439640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100b2b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41812/0 (socket says 172.21.15.161:41812) 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f2116439640 1 -- 172.21.15.161:0/3928355160 learned_addr learned my addr 172.21.15.161:0/3928355160 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:50.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f2115c38640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21100d08e0 0x7f2110153ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.299+0000 7f2116c3a640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f211015e240 0x7f21100cb3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2116439640 1 -- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f211015e240 msgr2=0x7f21100cb3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2116439640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f211015e240 0x7f21100cb3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2116439640 1 -- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21100d08e0 msgr2=0x7f2110153ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2116439640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21100d08e0 0x7f2110153ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2116439640 1 -- 172.21.15.161:0/3928355160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f21080094d0 con 0x7f21100ce4f0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2116439640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100b2b50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f210c00c7c0 tx=0x7f210c00cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2106ffd640 1 -- 172.21.15.161:0/3928355160 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c00a640 con 0x7f21100ce4f0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2106ffd640 1 -- 172.21.15.161:0/3928355160 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f210c00a7c0 con 0x7f21100ce4f0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f2106ffd640 1 -- 172.21.15.161:0/3928355160 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f210c00ab70 con 0x7f21100ce4f0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f21100b2350 con 0x7f21100ce4f0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.300+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f21100b3090 con 0x7f21100ce4f0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.301+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f21100b2870 con 0x7f21100ce4f0 2024-03-23T14:35:50.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.301+0000 7f2106ffd640 1 -- 172.21.15.161:0/3928355160 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f210c00acd0 con 0x7f21100ce4f0 2024-03-23T14:35:50.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.302+0000 7f2106ffd640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20f406b810 0x7f20f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.302+0000 7f2106ffd640 1 -- 172.21.15.161:0/3928355160 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f210c089920 con 0x7f21100ce4f0 2024-03-23T14:35:50.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.302+0000 7f2115c38640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20f406b810 0x7f20f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.302+0000 7f2115c38640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20f406b810 0x7f20f406dcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2110154820 tx=0x7f2100008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.306+0000 7f2106ffd640 1 -- 172.21.15.161:0/3928355160 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f210c054480 con 0x7f21100ce4f0 2024-03-23T14:35:50.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.478+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f211009fd60 con 0x7f21100ce4f0 2024-03-23T14:35:50.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.479+0000 7f2106ffd640 1 -- 172.21.15.161:0/3928355160 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7f210c018020 con 0x7f21100ce4f0 2024-03-23T14:35:50.476 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:50.476 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.481+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20f406b810 msgr2=0x7f20f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.481+0000 7f211869d640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20f406b810 0x7f20f406dcd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f2110154820 tx=0x7f2100008040 comp rx=0 tx=0).stop 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.481+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 msgr2=0x7f21100b2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.481+0000 7f211869d640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100b2b50 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f210c00c7c0 tx=0x7f210c00cc80 comp rx=0 tx=0).stop 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.482+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 shutdown_connections 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.482+0000 7f211869d640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20f406b810 0x7f20f406dcd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.482+0000 7f211869d640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f211015e240 0x7f21100cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.482+0000 7f211869d640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f21100d08e0 0x7f2110153ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.482+0000 7f211869d640 1 --2- 172.21.15.161:0/3928355160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f21100ce4f0 0x7f21100b2b50 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.482+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 >> 172.21.15.161:0/3928355160 conn(0x7f211009aa40 msgr2=0x7f21100cdaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:50.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.482+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 shutdown_connections 2024-03-23T14:35:50.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.482+0000 7f211869d640 1 -- 172.21.15.161:0/3928355160 wait complete. 2024-03-23T14:35:50.488 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:50.489 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:50.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.596+0000 7fb157ebb640 1 Processor -- start 2024-03-23T14:35:50.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.597+0000 7fb157ebb640 1 -- start start 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.597+0000 7fb157ebb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb150161820 0x7fb1500d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.597+0000 7fb157ebb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb1500d2ba0 0x7fb1500b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.597+0000 7fb157ebb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb150161c00 con 0x7fb1500d1270 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.597+0000 7fb157ebb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fb1500cea30 con 0x7fb1500d2ba0 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.597+0000 7fb157ebb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb1500cfd70 con 0x7fb150161820 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.597+0000 7fb155456640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb1500d1270 0x7fb1500d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37310/0 (socket says 172.21.15.161:37310) 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb155456640 1 -- 172.21.15.161:0/2526891767 learned_addr learned my addr 172.21.15.161:0/2526891767 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb155c57640 1 --2- 172.21.15.161:0/2526891767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb150161820 0x7fb1500d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb156458640 1 --2- 172.21.15.161:0/2526891767 >> v2:172.21.15.183:3300/0 conn(0x7fb1500d2ba0 0x7fb1500b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb154c55640 1 -- 172.21.15.161:0/2526891767 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3853352187 0 0) 0x7fb150161c00 con 0x7fb1500d1270 2024-03-23T14:35:50.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb154c55640 1 -- 172.21.15.161:0/2526891767 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb13c003520 con 0x7fb1500d1270 2024-03-23T14:35:50.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb155c57640 1 -- 172.21.15.161:0/2526891767 >> v2:172.21.15.183:3300/0 conn(0x7fb1500d2ba0 msgr2=0x7fb1500b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:50.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb155c57640 1 --2- 172.21.15.161:0/2526891767 >> v2:172.21.15.183:3300/0 conn(0x7fb1500d2ba0 0x7fb1500b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb155c57640 1 -- 172.21.15.161:0/2526891767 >> v1:172.21.15.161:6789/0 conn(0x7fb1500d1270 legacy=0x7fb1500d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb155c57640 1 -- 172.21.15.161:0/2526891767 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1500cecd0 con 0x7fb150161820 2024-03-23T14:35:50.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.598+0000 7fb156458640 1 --2- 172.21.15.161:0/2526891767 >> v2:172.21.15.183:3300/0 conn(0x7fb1500d2ba0 0x7fb1500b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:50.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb155c57640 1 --2- 172.21.15.161:0/2526891767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb150161820 0x7fb1500d47e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009d90 tx=0x7fb14c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4bf18e008fb9def server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb154c55640 1 -- 172.21.15.161:0/2526891767 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14c02cc50 con 0x7fb150161820 2024-03-23T14:35:50.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb154c55640 1 -- 172.21.15.161:0/2526891767 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb14c02cdb0 con 0x7fb150161820 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb154c55640 1 -- 172.21.15.161:0/2526891767 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb14c034640 con 0x7fb150161820 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb157ebb640 1 -- 172.21.15.161:0/2526891767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb150161820 msgr2=0x7fb1500d47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb157ebb640 1 --2- 172.21.15.161:0/2526891767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb150161820 0x7fb1500d47e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb14c009d90 tx=0x7fb14c02bea0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb157ebb640 1 -- 172.21.15.161:0/2526891767 shutdown_connections 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb157ebb640 1 --2- 172.21.15.161:0/2526891767 >> v2:172.21.15.183:3300/0 conn(0x7fb1500d2ba0 0x7fb1500b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb157ebb640 1 --2- 172.21.15.161:0/2526891767 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb150161820 0x7fb1500d47e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb157ebb640 1 -- 172.21.15.161:0/2526891767 >> 172.21.15.161:0/2526891767 conn(0x7fb15009d190 msgr2=0x7fb1500d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb157ebb640 1 -- 172.21.15.161:0/2526891767 shutdown_connections 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.599+0000 7fb157ebb640 1 -- 172.21.15.161:0/2526891767 wait complete. 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb157ebb640 1 Processor -- start 2024-03-23T14:35:50.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb157ebb640 1 -- start start 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb157ebb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1500d1270 0x7fb1500ca440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb157ebb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1500d2ba0 0x7fb1500c9290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb157ebb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb150161820 0x7fb150151080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb157ebb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb1500b6a30 con 0x7fb1500d2ba0 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb157ebb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb1500c81c0 con 0x7fb150161820 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb157ebb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb1500b6c80 con 0x7fb1500d1270 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb155456640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1500d2ba0 0x7fb1500c9290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb155456640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1500d2ba0 0x7fb1500c9290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57076/0 (socket says 172.21.15.161:57076) 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb155456640 1 -- 172.21.15.161:0/381258491 learned_addr learned my addr 172.21.15.161:0/381258491 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:50.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb155c57640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1500d1270 0x7fb1500ca440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.601+0000 7fb156458640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb150161820 0x7fb150151080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb155456640 1 -- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1500d1270 msgr2=0x7fb1500ca440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb155456640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1500d1270 0x7fb1500ca440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb155456640 1 -- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb150161820 msgr2=0x7fb150151080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb155456640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb150161820 0x7fb150151080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb155456640 1 -- 172.21.15.161:0/381258491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb148009400 con 0x7fb1500d2ba0 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb155456640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1500d2ba0 0x7fb1500c9290 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c460 tx=0x7fb14000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb1467fc640 1 -- 172.21.15.161:0/381258491 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb140005ce0 con 0x7fb1500d2ba0 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb1467fc640 1 -- 172.21.15.161:0/381258491 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb140005e40 con 0x7fb1500d2ba0 2024-03-23T14:35:50.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb14c009a70 con 0x7fb1500d2ba0 2024-03-23T14:35:50.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb1467fc640 1 -- 172.21.15.161:0/381258491 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1400072f0 con 0x7fb1500d2ba0 2024-03-23T14:35:50.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.602+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1500c7b70 con 0x7fb1500d2ba0 2024-03-23T14:35:50.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.603+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb15004f4e0 con 0x7fb1500d2ba0 2024-03-23T14:35:50.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.603+0000 7fb1467fc640 1 -- 172.21.15.161:0/381258491 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb140007450 con 0x7fb1500d2ba0 2024-03-23T14:35:50.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.604+0000 7fb1467fc640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb13406b840 0x7fb13406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.604+0000 7fb1467fc640 1 -- 172.21.15.161:0/381258491 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fb140089780 con 0x7fb1500d2ba0 2024-03-23T14:35:50.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.604+0000 7fb155c57640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb13406b840 0x7fb13406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.605+0000 7fb155c57640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb13406b840 0x7fb13406dd00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb14c036000 tx=0x7fb14c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.608+0000 7fb1467fc640 1 -- 172.21.15.161:0/381258491 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb1400541c0 con 0x7fb1500d2ba0 2024-03-23T14:35:50.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.780+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb1500d2270 con 0x7fb1500d2ba0 2024-03-23T14:35:50.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.781+0000 7fb1467fc640 1 -- 172.21.15.161:0/381258491 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7fb140017020 con 0x7fb1500d2ba0 2024-03-23T14:35:50.777 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb13406b840 msgr2=0x7fb13406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb13406b840 0x7fb13406dd00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fb14c036000 tx=0x7fb14c030000 comp rx=0 tx=0).stop 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1500d2ba0 msgr2=0x7fb1500c9290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1500d2ba0 0x7fb1500c9290 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fb14000c460 tx=0x7fb14000c860 comp rx=0 tx=0).stop 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 shutdown_connections 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb13406b840 0x7fb13406dd00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb150161820 0x7fb150151080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1500d2ba0 0x7fb1500c9290 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 --2- 172.21.15.161:0/381258491 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1500d1270 0x7fb1500ca440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 >> 172.21.15.161:0/381258491 conn(0x7fb15009d190 msgr2=0x7fb1500b1e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:50.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 shutdown_connections 2024-03-23T14:35:50.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.783+0000 7fb157ebb640 1 -- 172.21.15.161:0/381258491 wait complete. 2024-03-23T14:35:50.790 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:50.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.898+0000 7f7a444f4640 1 Processor -- start 2024-03-23T14:35:50.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.898+0000 7f7a444f4640 1 -- start start 2024-03-23T14:35:50.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a444f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c145520 0x7f7a3c160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a444f4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7a3c0cfee0 0x7f7a3c0d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a444f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7a3c070c20 0x7f7a3c0710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a444f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7a3c0d0880 con 0x7f7a3c070c20 2024-03-23T14:35:50.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a444f4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f7a3c1608c0 con 0x7f7a3c0cfee0 2024-03-23T14:35:50.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a444f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3c0ce410 con 0x7f7a3c145520 2024-03-23T14:35:50.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42a91640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7a3c070c20 0x7f7a3c0710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42a91640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7a3c070c20 0x7f7a3c0710b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57080/0 (socket says 172.21.15.161:57080) 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42a91640 1 -- 172.21.15.161:0/2972721552 learned_addr learned my addr 172.21.15.161:0/2972721552 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a41a8f640 1 --2- 172.21.15.161:0/2972721552 >> v2:172.21.15.183:3300/0 conn(0x7f7a3c0cfee0 0x7f7a3c0d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42290640 1 --2- 172.21.15.161:0/2972721552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c145520 0x7f7a3c160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42a91640 1 -- 172.21.15.161:0/2972721552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c145520 msgr2=0x7f7a3c160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42a91640 1 --2- 172.21.15.161:0/2972721552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c145520 0x7f7a3c160380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42a91640 1 -- 172.21.15.161:0/2972721552 >> v2:172.21.15.183:3300/0 conn(0x7f7a3c0cfee0 msgr2=0x7f7a3c0d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42a91640 1 --2- 172.21.15.161:0/2972721552 >> v2:172.21.15.183:3300/0 conn(0x7f7a3c0cfee0 0x7f7a3c0d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.899+0000 7f7a42a91640 1 -- 172.21.15.161:0/2972721552 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a3c0ce6f0 con 0x7f7a3c070c20 2024-03-23T14:35:50.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.900+0000 7f7a42a91640 1 --2- 172.21.15.161:0/2972721552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c070c20 0x7f7a3c0710b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c06c0d0 tx=0x7f7a3402be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df40e9f2c084965 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.900+0000 7f7a4128e640 1 -- 172.21.15.161:0/2972721552 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3402c9e0 con 0x7f7a3c070c20 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.900+0000 7f7a4128e640 1 -- 172.21.15.161:0/2972721552 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a3402cb40 con 0x7f7a3c070c20 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.900+0000 7f7a4128e640 1 -- 172.21.15.161:0/2972721552 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a340343f0 con 0x7f7a3c070c20 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 -- 172.21.15.161:0/2972721552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c070c20 msgr2=0x7f7a3c0710b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 --2- 172.21.15.161:0/2972721552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c070c20 0x7f7a3c0710b0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c06c0d0 tx=0x7f7a3402be40 comp rx=0 tx=0).stop 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 -- 172.21.15.161:0/2972721552 shutdown_connections 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 --2- 172.21.15.161:0/2972721552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c070c20 0x7f7a3c0710b0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 --2- 172.21.15.161:0/2972721552 >> v2:172.21.15.183:3300/0 conn(0x7f7a3c0cfee0 0x7f7a3c0d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 --2- 172.21.15.161:0/2972721552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c145520 0x7f7a3c160380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 -- 172.21.15.161:0/2972721552 >> 172.21.15.161:0/2972721552 conn(0x7f7a3c06b4a0 msgr2=0x7f7a3c06b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:50.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 -- 172.21.15.161:0/2972721552 shutdown_connections 2024-03-23T14:35:50.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.901+0000 7f7a444f4640 1 -- 172.21.15.161:0/2972721552 wait complete. 2024-03-23T14:35:50.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.902+0000 7f7a444f4640 1 Processor -- start 2024-03-23T14:35:50.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.902+0000 7f7a444f4640 1 -- start start 2024-03-23T14:35:50.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a444f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a3c070c20 0x7f7a3c13f800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a444f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c0cfee0 0x7f7a3c12d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a444f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c145520 0x7f7a3c12d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a444f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c12dc70 con 0x7f7a3c145520 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a444f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7a3c12df50 con 0x7f7a3c070c20 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a444f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3c12b070 con 0x7f7a3c0cfee0 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c145520 0x7f7a3c12d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c145520 0x7f7a3c12d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57092/0 (socket says 172.21.15.161:57092) 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 -- 172.21.15.161:0/156031724 learned_addr learned my addr 172.21.15.161:0/156031724 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42290640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a3c070c20 0x7f7a3c13f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a41a8f640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c0cfee0 0x7f7a3c12d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 -- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c0cfee0 msgr2=0x7f7a3c12d0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c0cfee0 0x7f7a3c12d0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 -- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a3c070c20 msgr2=0x7f7a3c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a3c070c20 0x7f7a3c13f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:50.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 -- 172.21.15.161:0/156031724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a34009b30 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a42a91640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c145520 0x7f7a3c12d5f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7a34036040 tx=0x7f7a34004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a32ffd640 1 -- 172.21.15.161:0/156031724 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a34009510 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a32ffd640 1 -- 172.21.15.161:0/156031724 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a3403e450 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a3c12b350 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a32ffd640 1 -- 172.21.15.161:0/156031724 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a3403e800 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.903+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a3c13e960 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.904+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a3c12b870 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.905+0000 7f7a32ffd640 1 -- 172.21.15.161:0/156031724 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7a3403e960 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.905+0000 7f7a32ffd640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a2006b810 0x7f7a2006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.905+0000 7f7a32ffd640 1 -- 172.21.15.161:0/156031724 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7a340acca0 con 0x7f7a3c145520 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.905+0000 7f7a42290640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a2006b810 0x7f7a2006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:50.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.906+0000 7f7a42290640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a2006b810 0x7f7a2006dcd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c144340 tx=0x7f7a38009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:50.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:50.910+0000 7f7a32ffd640 1 -- 172.21.15.161:0/156031724 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7a3c12b870 con 0x7f7a3c145520 2024-03-23T14:35:51.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.079+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7a3c12b870 con 0x7f7a3c145520 2024-03-23T14:35:51.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.079+0000 7f7a32ffd640 1 -- 172.21.15.161:0/156031724 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7f7a34043020 con 0x7f7a3c145520 2024-03-23T14:35:51.075 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.081+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a2006b810 msgr2=0x7f7a2006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.081+0000 7f7a444f4640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a2006b810 0x7f7a2006dcd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f7a3c144340 tx=0x7f7a38009190 comp rx=0 tx=0).stop 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.081+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c145520 msgr2=0x7f7a3c12d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.081+0000 7f7a444f4640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c145520 0x7f7a3c12d5f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f7a34036040 tx=0x7f7a34004a40 comp rx=0 tx=0).stop 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.082+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 shutdown_connections 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.082+0000 7f7a444f4640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a2006b810 0x7f7a2006dcd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.082+0000 7f7a444f4640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a3c145520 0x7f7a3c12d5f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.082+0000 7f7a444f4640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a3c0cfee0 0x7f7a3c12d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.082+0000 7f7a444f4640 1 --2- 172.21.15.161:0/156031724 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a3c070c20 0x7f7a3c13f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.082+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 >> 172.21.15.161:0/156031724 conn(0x7f7a3c06b4a0 msgr2=0x7f7a3c0b1fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:51.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.082+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 shutdown_connections 2024-03-23T14:35:51.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.082+0000 7f7a444f4640 1 -- 172.21.15.161:0/156031724 wait complete. 2024-03-23T14:35:51.088 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:51.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.195+0000 7f7d2bceb640 1 Processor -- start 2024-03-23T14:35:51.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.196+0000 7f7d2bceb640 1 -- start start 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.196+0000 7f7d2bceb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7d240b2aa0 0x7f7d240cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.196+0000 7f7d2bceb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d240d01b0 0x7f7d2406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.196+0000 7f7d2bceb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d240d0600 con 0x7f7d241552f0 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.196+0000 7f7d2bceb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7d2415f480 con 0x7f7d240b2aa0 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.196+0000 7f7d2bceb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7d241556d0 con 0x7f7d240d01b0 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d29286640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7d240b2aa0 0x7f7d240cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d29a87640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7d241552f0 0x7f7d240b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33048/0 (socket says 172.21.15.161:33048) 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d29a87640 1 -- 172.21.15.161:0/1441457711 learned_addr learned my addr 172.21.15.161:0/1441457711 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d2a288640 1 --2- 172.21.15.161:0/1441457711 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d240d01b0 0x7f7d2406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d28a85640 1 -- 172.21.15.161:0/1441457711 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254570438 0 0) 0x7f7d14001110 con 0x7f7d241552f0 2024-03-23T14:35:51.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d29286640 1 -- 172.21.15.161:0/1441457711 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d240d01b0 msgr2=0x7f7d2406ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:51.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d29286640 1 --2- 172.21.15.161:0/1441457711 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d240d01b0 0x7f7d2406ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d29286640 1 -- 172.21.15.161:0/1441457711 >> v1:172.21.15.183:6789/0 conn(0x7f7d241552f0 legacy=0x7f7d240b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d29286640 1 -- 172.21.15.161:0/1441457711 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d2415e870 con 0x7f7d240b2aa0 2024-03-23T14:35:51.194 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.197+0000 7f7d2a288640 1 --2- 172.21.15.161:0/1441457711 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d240d01b0 0x7f7d2406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:51.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d29286640 1 --2- 172.21.15.161:0/1441457711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240b2aa0 0x7f7d240cfc70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7d200097a0 tx=0x7f7d2002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13c9df7385b307d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d28a85640 1 -- 172.21.15.161:0/1441457711 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20031830 con 0x7f7d240b2aa0 2024-03-23T14:35:51.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d28a85640 1 -- 172.21.15.161:0/1441457711 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d20031990 con 0x7f7d240b2aa0 2024-03-23T14:35:51.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d28a85640 1 -- 172.21.15.161:0/1441457711 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d20031d20 con 0x7f7d240b2aa0 2024-03-23T14:35:51.195 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d2bceb640 1 -- 172.21.15.161:0/1441457711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240b2aa0 msgr2=0x7f7d240cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/1441457711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240b2aa0 0x7f7d240cfc70 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f7d200097a0 tx=0x7f7d2002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d2bceb640 1 -- 172.21.15.161:0/1441457711 shutdown_connections 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/1441457711 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d240d01b0 0x7f7d2406ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/1441457711 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240b2aa0 0x7f7d240cfc70 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d2bceb640 1 -- 172.21.15.161:0/1441457711 >> 172.21.15.161:0/1441457711 conn(0x7f7d2406b260 msgr2=0x7f7d2406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d2bceb640 1 -- 172.21.15.161:0/1441457711 shutdown_connections 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.198+0000 7f7d2bceb640 1 -- 172.21.15.161:0/1441457711 wait complete. 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d2bceb640 1 Processor -- start 2024-03-23T14:35:51.196 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d2bceb640 1 -- start start 2024-03-23T14:35:51.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d2bceb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d240b2aa0 0x7f7d24145060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d2bceb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240d01b0 0x7f7d24148870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d2bceb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d241552f0 0x7f7d24153c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d2bceb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7d240726e0 con 0x7f7d240d01b0 2024-03-23T14:35:51.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d2bceb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7d241455a0 con 0x7f7d240b2aa0 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d2bceb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7d24156130 con 0x7f7d241552f0 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d29286640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240d01b0 0x7f7d24148870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d29286640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240d01b0 0x7f7d24148870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57102/0 (socket says 172.21.15.161:57102) 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d29286640 1 -- 172.21.15.161:0/3997396590 learned_addr learned my addr 172.21.15.161:0/3997396590 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.200+0000 7f7d29a87640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d240b2aa0 0x7f7d24145060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d2a288640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d241552f0 0x7f7d24153c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d29286640 1 -- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d241552f0 msgr2=0x7f7d24153c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d29286640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d241552f0 0x7f7d24153c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d29286640 1 -- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d240b2aa0 msgr2=0x7f7d24145060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d29286640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d240b2aa0 0x7f7d24145060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d29286640 1 -- 172.21.15.161:0/3997396590 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1c009a70 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d29286640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240d01b0 0x7f7d24148870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7d2002bee0 tx=0x7f7d200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d1a7fc640 1 -- 172.21.15.161:0/3997396590 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2003a630 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d1a7fc640 1 -- 172.21.15.161:0/3997396590 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2003a7b0 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d20009400 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d1a7fc640 1 -- 172.21.15.161:0/3997396590 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d2003ab60 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.201+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d241566f0 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.202+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d2404f4e0 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.202+0000 7f7d1a7fc640 1 -- 172.21.15.161:0/3997396590 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7d2003acc0 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.203+0000 7f7d1a7fc640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7d0806b810 0x7f7d0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.203+0000 7f7d1a7fc640 1 -- 172.21.15.161:0/3997396590 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f7d200abc50 con 0x7f7d240d01b0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.203+0000 7f7d29a87640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7d0806b810 0x7f7d0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.203+0000 7f7d29a87640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7d0806b810 0x7f7d0806dcd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7d14000f60 tx=0x7f7d14009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.202 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.207+0000 7f7d1a7fc640 1 -- 172.21.15.161:0/3997396590 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7d200b1020 con 0x7f7d240d01b0 2024-03-23T14:35:51.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.378+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f7d2406d7a0 con 0x7f7d240d01b0 2024-03-23T14:35:51.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.379+0000 7f7d1a7fc640 1 -- 172.21.15.161:0/3997396590 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v20) ==== 100+0+11 (secure 0 0 0) 0x7f7d20076840 con 0x7f7d240d01b0 2024-03-23T14:35:51.375 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7d0806b810 msgr2=0x7f7d0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7d0806b810 0x7f7d0806dcd0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f7d14000f60 tx=0x7f7d14009040 comp rx=0 tx=0).stop 2024-03-23T14:35:51.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240d01b0 msgr2=0x7f7d24148870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240d01b0 0x7f7d24148870 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7d2002bee0 tx=0x7f7d200046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 shutdown_connections 2024-03-23T14:35:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7d0806b810 0x7f7d0806dcd0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7d241552f0 0x7f7d24153c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.378 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7d240d01b0 0x7f7d24148870 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 --2- 172.21.15.161:0/3997396590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7d240b2aa0 0x7f7d24145060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 >> 172.21.15.161:0/3997396590 conn(0x7f7d2406b260 msgr2=0x7f7d2415b2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:51.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 shutdown_connections 2024-03-23T14:35:51.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.381+0000 7f7d2bceb640 1 -- 172.21.15.161:0/3997396590 wait complete. 2024-03-23T14:35:51.388 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:35:51.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.495+0000 7fa44e11e640 1 Processor -- start 2024-03-23T14:35:51.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.495+0000 7fa44e11e640 1 -- start start 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa44e11e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa448072100 0x7fa44815b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa44e11e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa4480724e0 0x7fa44806e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa44e11e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa44815be70 con 0x7fa44815b930 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa44e11e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa44815c430 con 0x7fa4480724e0 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa44e11e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa44815c150 con 0x7fa448072100 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa446ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa4480724e0 0x7fa44806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa446ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa4480724e0 0x7fa44806e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:54384/0 (socket says 172.21.15.161:54384) 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa446ffd640 1 -- 172.21.15.161:0/2115980299 learned_addr learned my addr 172.21.15.161:0/2115980299 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa4477fe640 1 --2- 172.21.15.161:0/2115980299 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa448072100 0x7fa44815b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa446ffd640 1 -- 172.21.15.161:0/2115980299 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa448072100 msgr2=0x7fa44815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa446ffd640 1 --2- 172.21.15.161:0/2115980299 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa448072100 0x7fa44815b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa446ffd640 1 -- 172.21.15.161:0/2115980299 >> v1:172.21.15.183:6789/0 conn(0x7fa44815b930 legacy=0x7fa44815e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.496+0000 7fa446ffd640 1 -- 172.21.15.161:0/2115980299 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4480b2c70 con 0x7fa4480724e0 2024-03-23T14:35:51.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.497+0000 7fa446ffd640 1 --2- 172.21.15.161:0/2115980299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4480724e0 0x7fa44806e590 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009870 tx=0x7fa43c02b980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8c7387fbb835443 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.497+0000 7fa4467fc640 1 -- 172.21.15.161:0/2115980299 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c030840 con 0x7fa4480724e0 2024-03-23T14:35:51.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.497+0000 7fa4467fc640 1 -- 172.21.15.161:0/2115980299 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa43c0309a0 con 0x7fa4480724e0 2024-03-23T14:35:51.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.497+0000 7fa4467fc640 1 -- 172.21.15.161:0/2115980299 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa43c030d30 con 0x7fa4480724e0 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.498+0000 7fa44e11e640 1 -- 172.21.15.161:0/2115980299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4480724e0 msgr2=0x7fa44806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.498+0000 7fa44e11e640 1 --2- 172.21.15.161:0/2115980299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4480724e0 0x7fa44806e590 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009870 tx=0x7fa43c02b980 comp rx=0 tx=0).stop 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.498+0000 7fa44e11e640 1 -- 172.21.15.161:0/2115980299 shutdown_connections 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.498+0000 7fa44e11e640 1 --2- 172.21.15.161:0/2115980299 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4480724e0 0x7fa44806e590 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.498+0000 7fa44e11e640 1 --2- 172.21.15.161:0/2115980299 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa448072100 0x7fa44815b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.498+0000 7fa44e11e640 1 -- 172.21.15.161:0/2115980299 >> 172.21.15.161:0/2115980299 conn(0x7fa4480b17f0 msgr2=0x7fa4480b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.498+0000 7fa44e11e640 1 -- 172.21.15.161:0/2115980299 shutdown_connections 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.498+0000 7fa44e11e640 1 -- 172.21.15.161:0/2115980299 wait complete. 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.499+0000 7fa44e11e640 1 Processor -- start 2024-03-23T14:35:51.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.499+0000 7fa44e11e640 1 -- start start 2024-03-23T14:35:51.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa44e11e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa448072100 0x7fa44813f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa44e11e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4480724e0 0x7fa4481e5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa44e11e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa44815b930 0x7fa4481e5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa44e11e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa448140640 con 0x7fa44815b930 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa44e11e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa44813fad0 con 0x7fa448072100 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa44e11e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa4481e66c0 con 0x7fa4480724e0 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa44815b930 0x7fa4481e5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa44815b930 0x7fa4481e5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57114/0 (socket says 172.21.15.161:57114) 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 -- 172.21.15.161:0/12376464 learned_addr learned my addr 172.21.15.161:0/12376464 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa446ffd640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4480724e0 0x7fa4481e5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa4477fe640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa448072100 0x7fa44813f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 -- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4480724e0 msgr2=0x7fa4481e5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4480724e0 0x7fa4481e5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 -- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa448072100 msgr2=0x7fa44813f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa448072100 0x7fa44813f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 -- 172.21.15.161:0/12376464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa43c0094d0 con 0x7fa44815b930 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa447fff640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa44815b930 0x7fa4481e5ba0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa44000e940 tx=0x7fa44000ee00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.500+0000 7fa42ffff640 1 -- 172.21.15.161:0/12376464 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440001e90 con 0x7fa44815b930 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.501+0000 7fa42ffff640 1 -- 172.21.15.161:0/12376464 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa440001ff0 con 0x7fa44815b930 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.501+0000 7fa42ffff640 1 -- 172.21.15.161:0/12376464 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa440003bb0 con 0x7fa44815b930 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.501+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4481e63e0 con 0x7fa44815b930 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.501+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa448140260 con 0x7fa44815b930 2024-03-23T14:35:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.502+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa448140780 con 0x7fa44815b930 2024-03-23T14:35:51.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.502+0000 7fa42ffff640 1 -- 172.21.15.161:0/12376464 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa440003d10 con 0x7fa44815b930 2024-03-23T14:35:51.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.503+0000 7fa42ffff640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa43406b810 0x7fa43406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.503+0000 7fa42ffff640 1 -- 172.21.15.161:0/12376464 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa44008c580 con 0x7fa44815b930 2024-03-23T14:35:51.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.503+0000 7fa4477fe640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa43406b810 0x7fa43406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.503+0000 7fa4477fe640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa43406b810 0x7fa43406dcd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa4480b0ee0 tx=0x7fa438009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.507+0000 7fa42ffff640 1 -- 172.21.15.161:0/12376464 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa448140780 con 0x7fa44815b930 2024-03-23T14:35:51.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.689+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fa448140780 con 0x7fa44815b930 2024-03-23T14:35:51.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.689+0000 7fa42ffff640 1 -- 172.21.15.161:0/12376464 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v20) ==== 80+0+0 (secure 0 0 0) 0x7fa448140780 con 0x7fa44815b930 2024-03-23T14:35:51.685 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa43406b810 msgr2=0x7fa43406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa43406b810 0x7fa43406dcd0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa4480b0ee0 tx=0x7fa438009190 comp rx=0 tx=0).stop 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa44815b930 msgr2=0x7fa4481e5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa44815b930 0x7fa4481e5ba0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa44000e940 tx=0x7fa44000ee00 comp rx=0 tx=0).stop 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 shutdown_connections 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa43406b810 0x7fa43406dcd0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa44815b930 0x7fa4481e5ba0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa4480724e0 0x7fa4481e5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 --2- 172.21.15.161:0/12376464 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa448072100 0x7fa44813f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.691+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 >> 172.21.15.161:0/12376464 conn(0x7fa4480b17f0 msgr2=0x7fa4480b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.692+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 shutdown_connections 2024-03-23T14:35:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.692+0000 7fa44e11e640 1 -- 172.21.15.161:0/12376464 wait complete. 2024-03-23T14:35:51.698 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:35:51.699 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:35:51.715 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:51.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.819+0000 7eff7f1d7640 1 Processor -- start 2024-03-23T14:35:51.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.819+0000 7eff7f1d7640 1 -- start start 2024-03-23T14:35:51.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff7f1d7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7eff780cc5f0 0x7eff780c8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff7f1d7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff7802c050 con 0x7eff7815bff0 2024-03-23T14:35:51.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff7f1d7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff7802c2f0 con 0x7eff780c8bb0 2024-03-23T14:35:51.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff7f1d7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7eff780b2980 con 0x7eff780cc5f0 2024-03-23T14:35:51.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff7cf73640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7eff7815bff0 0x7eff780cbee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37314/0 (socket says 172.21.15.161:37314) 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff7cf73640 1 -- 172.21.15.161:0/2619312819 learned_addr learned my addr 172.21.15.161:0/2619312819 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff77fff640 1 --2- 172.21.15.161:0/2619312819 >> v2:172.21.15.183:3300/0 conn(0x7eff780cc5f0 0x7eff780c8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff777fe640 1 -- 172.21.15.161:0/2619312819 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1049754918 0 0) 0x7eff7802c050 con 0x7eff7815bff0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.820+0000 7eff777fe640 1 -- 172.21.15.161:0/2619312819 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff60003520 con 0x7eff7815bff0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.821+0000 7eff777fe640 1 -- 172.21.15.161:0/2619312819 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 373185323 0 0) 0x7eff7802c2f0 con 0x7eff780c8bb0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.821+0000 7eff777fe640 1 -- 172.21.15.161:0/2619312819 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff7802c050 con 0x7eff780c8bb0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.821+0000 7eff77fff640 1 -- 172.21.15.161:0/2619312819 >> v1:172.21.15.183:6789/0 conn(0x7eff780c8bb0 legacy=0x7eff780c9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.821+0000 7eff77fff640 1 -- 172.21.15.161:0/2619312819 >> v1:172.21.15.161:6789/0 conn(0x7eff7815bff0 legacy=0x7eff780cbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.821+0000 7eff77fff640 1 -- 172.21.15.161:0/2619312819 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7802c5d0 con 0x7eff780cc5f0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.821+0000 7eff77fff640 1 --2- 172.21.15.161:0/2619312819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 0x7eff780c8670 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7eff780b2ac0 tx=0x7eff6c02b940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f98ce1c93a4732dc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff777fe640 1 -- 172.21.15.161:0/2619312819 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c030800 con 0x7eff780cc5f0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff777fe640 1 -- 172.21.15.161:0/2619312819 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff6c030960 con 0x7eff780cc5f0 2024-03-23T14:35:51.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff777fe640 1 -- 172.21.15.161:0/2619312819 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c030cf0 con 0x7eff780cc5f0 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2619312819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 msgr2=0x7eff780c8670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff7f1d7640 1 --2- 172.21.15.161:0/2619312819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 0x7eff780c8670 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7eff780b2ac0 tx=0x7eff6c02b940 comp rx=0 tx=0).stop 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2619312819 shutdown_connections 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff7f1d7640 1 --2- 172.21.15.161:0/2619312819 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 0x7eff780c8670 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2619312819 >> 172.21.15.161:0/2619312819 conn(0x7eff780b1e60 msgr2=0x7eff780b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2619312819 shutdown_connections 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.822+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2619312819 wait complete. 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.823+0000 7eff7f1d7640 1 Processor -- start 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.823+0000 7eff7f1d7640 1 -- start start 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7f1d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff780c8bb0 0x7eff7815a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7f1d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 0x7eff780b0810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7f1d7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff7815bff0 0x7eff7815e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7f1d7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff780ad470 con 0x7eff7815bff0 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7f1d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7eff780ad750 con 0x7eff780cc5f0 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7f1d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7eff780ada30 con 0x7eff780c8bb0 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7d774640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff7815bff0 0x7eff7815e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7d774640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff7815bff0 0x7eff7815e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57116/0 (socket says 172.21.15.161:57116) 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.824+0000 7eff7d774640 1 -- 172.21.15.161:0/2712829742 learned_addr learned my addr 172.21.15.161:0/2712829742 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff77fff640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 0x7eff780b0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7cf73640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff780c8bb0 0x7eff7815a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7d774640 1 -- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff780c8bb0 msgr2=0x7eff7815a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7d774640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff780c8bb0 0x7eff7815a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7d774640 1 -- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 msgr2=0x7eff780b0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7d774640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 0x7eff780b0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7d774640 1 -- 172.21.15.161:0/2712829742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6c009400 con 0x7eff7815bff0 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7d774640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff7815bff0 0x7eff7815e5a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7eff7000c790 tx=0x7eff7000cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff757fa640 1 -- 172.21.15.161:0/2712829742 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7000cea0 con 0x7eff7815bff0 2024-03-23T14:35:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff757fa640 1 -- 172.21.15.161:0/2712829742 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff70003b70 con 0x7eff7815bff0 2024-03-23T14:35:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff757fa640 1 -- 172.21.15.161:0/2712829742 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7000a850 con 0x7eff7815bff0 2024-03-23T14:35:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff780add10 con 0x7eff7815bff0 2024-03-23T14:35:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.825+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff780ad140 con 0x7eff7815bff0 2024-03-23T14:35:51.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.826+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff780ae230 con 0x7eff7815bff0 2024-03-23T14:35:51.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.827+0000 7eff757fa640 1 -- 172.21.15.161:0/2712829742 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7eff7000a9b0 con 0x7eff7815bff0 2024-03-23T14:35:51.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.827+0000 7eff757fa640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6406b810 0x7eff6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:51.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.827+0000 7eff757fa640 1 -- 172.21.15.161:0/2712829742 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7eff70089f70 con 0x7eff7815bff0 2024-03-23T14:35:51.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.827+0000 7eff7cf73640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6406b810 0x7eff6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:51.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.828+0000 7eff7cf73640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6406b810 0x7eff6406dcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7eff680011e0 tx=0x7eff68007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:51.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:51.831+0000 7eff757fa640 1 -- 172.21.15.161:0/2712829742 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7eff700bde10 con 0x7eff7815bff0 2024-03-23T14:35:51.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.003+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7eff78067aa0 con 0x7eff7815bff0 2024-03-23T14:35:52.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.004+0000 7eff757fa640 1 -- 172.21.15.161:0/2712829742 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16641 (secure 0 0 0) 0x7eff780ae230 con 0x7eff7815bff0 2024-03-23T14:35:52.000 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:52.000 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":20,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:14.765618+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:52.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6406b810 msgr2=0x7eff6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6406b810 0x7eff6406dcd0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7eff680011e0 tx=0x7eff68007d30 comp rx=0 tx=0).stop 2024-03-23T14:35:52.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff7815bff0 msgr2=0x7eff7815e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff7815bff0 0x7eff7815e5a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7eff7000c790 tx=0x7eff7000cc50 comp rx=0 tx=0).stop 2024-03-23T14:35:52.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 shutdown_connections 2024-03-23T14:35:52.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7eff6406b810 0x7eff6406dcd0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff7815bff0 0x7eff7815e5a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff780cc5f0 0x7eff780b0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 --2- 172.21.15.161:0/2712829742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff780c8bb0 0x7eff7815a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.006+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 >> 172.21.15.161:0/2712829742 conn(0x7eff780b1e60 msgr2=0x7eff7806e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:52.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.007+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 shutdown_connections 2024-03-23T14:35:52.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.007+0000 7eff7f1d7640 1 -- 172.21.15.161:0/2712829742 wait complete. 2024-03-23T14:35:52.014 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:52.014 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.122+0000 7f07b6c15640 1 Processor -- start 2024-03-23T14:35:52.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.122+0000 7f07b6c15640 1 -- start start 2024-03-23T14:35:52.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07b6c15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00b2980 0x7f07b00cced0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07b6c15640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f07b00cd410 0x7f07b015fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07b6c15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f07b00c8480 con 0x7f07b015bfd0 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07b6c15640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f07b002c1c0 con 0x7f07b00cd410 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07b6c15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f07b002c650 con 0x7f07b00b2980 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07b49b1640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f07b015bfd0 0x7f07b00c7d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33070/0 (socket says 172.21.15.161:33070) 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07b49b1640 1 -- 172.21.15.161:0/1092480952 learned_addr learned my addr 172.21.15.161:0/1092480952 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07b51b2640 1 --2- 172.21.15.161:0/1092480952 >> v2:172.21.15.183:3300/0 conn(0x7f07b00cd410 0x7f07b015fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07affff640 1 --2- 172.21.15.161:0/1092480952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00b2980 0x7f07b00cced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.123+0000 7f07af7fe640 1 -- 172.21.15.161:0/1092480952 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3536323061 0 0) 0x7f07b00c8480 con 0x7f07b015bfd0 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.124+0000 7f07af7fe640 1 -- 172.21.15.161:0/1092480952 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0798003520 con 0x7f07b015bfd0 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.124+0000 7f07affff640 1 -- 172.21.15.161:0/1092480952 >> v2:172.21.15.183:3300/0 conn(0x7f07b00cd410 msgr2=0x7f07b015fc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.124+0000 7f07affff640 1 --2- 172.21.15.161:0/1092480952 >> v2:172.21.15.183:3300/0 conn(0x7f07b00cd410 0x7f07b015fc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.124+0000 7f07affff640 1 -- 172.21.15.161:0/1092480952 >> v1:172.21.15.183:6789/0 conn(0x7f07b015bfd0 legacy=0x7f07b00c7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.124+0000 7f07affff640 1 -- 172.21.15.161:0/1092480952 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07b00c3b90 con 0x7f07b00b2980 2024-03-23T14:35:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.124+0000 7f07affff640 1 --2- 172.21.15.161:0/1092480952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00b2980 0x7f07b00cced0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f07a40097a0 tx=0x7f07a402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fd472d9fa2efe630 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.124+0000 7f07af7fe640 1 -- 172.21.15.161:0/1092480952 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4031830 con 0x7f07b00b2980 2024-03-23T14:35:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07af7fe640 1 -- 172.21.15.161:0/1092480952 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a4031990 con 0x7f07b00b2980 2024-03-23T14:35:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07af7fe640 1 -- 172.21.15.161:0/1092480952 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a4031d20 con 0x7f07b00b2980 2024-03-23T14:35:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07b6c15640 1 -- 172.21.15.161:0/1092480952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00b2980 msgr2=0x7f07b00cced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07b6c15640 1 --2- 172.21.15.161:0/1092480952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00b2980 0x7f07b00cced0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f07a40097a0 tx=0x7f07a402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07b6c15640 1 -- 172.21.15.161:0/1092480952 shutdown_connections 2024-03-23T14:35:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07b6c15640 1 --2- 172.21.15.161:0/1092480952 >> v2:172.21.15.183:3300/0 conn(0x7f07b00cd410 0x7f07b015fc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07b6c15640 1 --2- 172.21.15.161:0/1092480952 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00b2980 0x7f07b00cced0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07b6c15640 1 -- 172.21.15.161:0/1092480952 >> 172.21.15.161:0/1092480952 conn(0x7f07b00b1e60 msgr2=0x7f07b00b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07b6c15640 1 -- 172.21.15.161:0/1092480952 shutdown_connections 2024-03-23T14:35:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.125+0000 7f07b6c15640 1 -- 172.21.15.161:0/1092480952 wait complete. 2024-03-23T14:35:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.126+0000 7f07b6c15640 1 Processor -- start 2024-03-23T14:35:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.126+0000 7f07b6c15640 1 -- start start 2024-03-23T14:35:52.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b6c15640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07b00b2980 0x7f07b00c2c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b6c15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00cd410 0x7f07b009a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b6c15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07b015bfd0 0x7f07b009c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b6c15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f07b00cdfe0 con 0x7f07b015bfd0 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b6c15640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f07b00ce2c0 con 0x7f07b00b2980 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b6c15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f07b00ce5a0 con 0x7f07b00cd410 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b51b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07b015bfd0 0x7f07b009c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b51b2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07b015bfd0 0x7f07b009c130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57122/0 (socket says 172.21.15.161:57122) 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b51b2640 1 -- 172.21.15.161:0/465329620 learned_addr learned my addr 172.21.15.161:0/465329620 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:52.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07affff640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00cd410 0x7f07b009a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.127+0000 7f07b49b1640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07b00b2980 0x7f07b00c2c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07b51b2640 1 -- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00cd410 msgr2=0x7f07b009a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07b51b2640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00cd410 0x7f07b009a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07b51b2640 1 -- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07b00b2980 msgr2=0x7f07b00c2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07b51b2640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07b00b2980 0x7f07b00c2c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07b51b2640 1 -- 172.21.15.161:0/465329620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a4009400 con 0x7f07b015bfd0 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07b51b2640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07b015bfd0 0x7f07b009c130 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f07a800efc0 tx=0x7f07a800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07ad7fa640 1 -- 172.21.15.161:0/465329620 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a8009ca0 con 0x7f07b015bfd0 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07ad7fa640 1 -- 172.21.15.161:0/465329620 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f07a800f040 con 0x7f07b015bfd0 2024-03-23T14:35:52.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07b00ce880 con 0x7f07b015bfd0 2024-03-23T14:35:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07ad7fa640 1 -- 172.21.15.161:0/465329620 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f07a80146c0 con 0x7f07b015bfd0 2024-03-23T14:35:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.128+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07b009aa70 con 0x7f07b015bfd0 2024-03-23T14:35:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.129+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07b00ceac0 con 0x7f07b015bfd0 2024-03-23T14:35:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.129+0000 7f07ad7fa640 1 -- 172.21.15.161:0/465329620 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f07a8014820 con 0x7f07b015bfd0 2024-03-23T14:35:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.130+0000 7f07ad7fa640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f079c06b810 0x7f079c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.130+0000 7f07ad7fa640 1 -- 172.21.15.161:0/465329620 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f07a808b080 con 0x7f07b015bfd0 2024-03-23T14:35:52.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.130+0000 7f07b49b1640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f079c06b810 0x7f079c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.130+0000 7f07b49b1640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f079c06b810 0x7f079c06dcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f07a0005880 tx=0x7f07a0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.134+0000 7f07ad7fa640 1 -- 172.21.15.161:0/465329620 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f07b00ceac0 con 0x7f07b015bfd0 2024-03-23T14:35:52.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.305+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f07b00ceac0 con 0x7f07b015bfd0 2024-03-23T14:35:52.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.305+0000 7f07ad7fa640 1 -- 172.21.15.161:0/465329620 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (secure 0 0 0) 0x7f07b00ceac0 con 0x7f07b015bfd0 2024-03-23T14:35:52.301 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:52.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.307+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f079c06b810 msgr2=0x7f079c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.307+0000 7f07b6c15640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f079c06b810 0x7f079c06dcd0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f07a0005880 tx=0x7f07a0009040 comp rx=0 tx=0).stop 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07b015bfd0 msgr2=0x7f07b009c130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07b015bfd0 0x7f07b009c130 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f07a800efc0 tx=0x7f07a800c490 comp rx=0 tx=0).stop 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 shutdown_connections 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f079c06b810 0x7f079c06dcd0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f07b015bfd0 0x7f07b009c130 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f07b00cd410 0x7f07b009a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 --2- 172.21.15.161:0/465329620 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f07b00b2980 0x7f07b00c2c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 >> 172.21.15.161:0/465329620 conn(0x7f07b00b1e60 msgr2=0x7f07b00b2240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 shutdown_connections 2024-03-23T14:35:52.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.308+0000 7f07b6c15640 1 -- 172.21.15.161:0/465329620 wait complete. 2024-03-23T14:35:52.314 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:52.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.422+0000 7efe2f2a4640 1 Processor -- start 2024-03-23T14:35:52.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.422+0000 7efe2f2a4640 1 -- start start 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.423+0000 7efe2f2a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe280cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.423+0000 7efe2f2a4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe2802bfb0 con 0x7efe2815e2a0 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.423+0000 7efe2f2a4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efe280cdf00 con 0x7efe280cd1c0 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.423+0000 7efe2f2a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efe280ce1c0 con 0x7efe280cb2d0 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.423+0000 7efe2d040640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efe2815e2a0 0x7efe280ccab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37326/0 (socket says 172.21.15.161:37326) 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.423+0000 7efe2d040640 1 -- 172.21.15.161:0/3202115779 learned_addr learned my addr 172.21.15.161:0/3202115779 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.423+0000 7efe2d841640 1 --2- 172.21.15.161:0/3202115779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe280cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.424+0000 7efe1ffff640 1 -- 172.21.15.161:0/3202115779 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1143272453 0 0) 0x7efe2802bfb0 con 0x7efe2815e2a0 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.424+0000 7efe1ffff640 1 -- 172.21.15.161:0/3202115779 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe14003520 con 0x7efe2815e2a0 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.424+0000 7efe1ffff640 1 -- 172.21.15.161:0/3202115779 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 74425399 0 0) 0x7efe280cdf00 con 0x7efe280cd1c0 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.424+0000 7efe1ffff640 1 -- 172.21.15.161:0/3202115779 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efe2802bfb0 con 0x7efe280cd1c0 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.424+0000 7efe2d841640 1 -- 172.21.15.161:0/3202115779 >> v1:172.21.15.183:6789/0 conn(0x7efe280cd1c0 legacy=0x7efe280cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.424+0000 7efe2d841640 1 -- 172.21.15.161:0/3202115779 >> v1:172.21.15.161:6789/0 conn(0x7efe2815e2a0 legacy=0x7efe280ccab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.424+0000 7efe2d841640 1 -- 172.21.15.161:0/3202115779 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe280cc690 con 0x7efe280cb2d0 2024-03-23T14:35:52.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe2d841640 1 --2- 172.21.15.161:0/3202115779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe280cb760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efe200097d0 tx=0x7efe2002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=856c34b0f0402291 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe1ffff640 1 -- 172.21.15.161:0/3202115779 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20031770 con 0x7efe280cb2d0 2024-03-23T14:35:52.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe1ffff640 1 -- 172.21.15.161:0/3202115779 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe200318d0 con 0x7efe280cb2d0 2024-03-23T14:35:52.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe1ffff640 1 -- 172.21.15.161:0/3202115779 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20031c60 con 0x7efe280cb2d0 2024-03-23T14:35:52.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe2f2a4640 1 -- 172.21.15.161:0/3202115779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 msgr2=0x7efe280cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe2f2a4640 1 --2- 172.21.15.161:0/3202115779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe280cb760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7efe200097d0 tx=0x7efe2002b810 comp rx=0 tx=0).stop 2024-03-23T14:35:52.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe2f2a4640 1 -- 172.21.15.161:0/3202115779 shutdown_connections 2024-03-23T14:35:52.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe2f2a4640 1 --2- 172.21.15.161:0/3202115779 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe280cb760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe2f2a4640 1 -- 172.21.15.161:0/3202115779 >> 172.21.15.161:0/3202115779 conn(0x7efe280b1e60 msgr2=0x7efe280b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:52.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe2f2a4640 1 -- 172.21.15.161:0/3202115779 shutdown_connections 2024-03-23T14:35:52.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.425+0000 7efe2f2a4640 1 -- 172.21.15.161:0/3202115779 wait complete. 2024-03-23T14:35:52.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.426+0000 7efe2f2a4640 1 Processor -- start 2024-03-23T14:35:52.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2f2a4640 1 -- start start 2024-03-23T14:35:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2f2a4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe281f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2f2a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe280cd1c0 0x7efe28156960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2f2a4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe2815e2a0 0x7efe28156ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2f2a4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efe281580e0 con 0x7efe280cd1c0 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2f2a4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efe281574d0 con 0x7efe2815e2a0 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2f2a4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efe281577b0 con 0x7efe280cb2d0 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2c83f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe280cd1c0 0x7efe28156960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2c83f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe280cd1c0 0x7efe28156960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57134/0 (socket says 172.21.15.161:57134) 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2c83f640 1 -- 172.21.15.161:0/2831715549 learned_addr learned my addr 172.21.15.161:0/2831715549 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2d841640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe2815e2a0 0x7efe28156ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.427+0000 7efe2d040640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe281f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe2c83f640 1 -- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 msgr2=0x7efe281f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe2c83f640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe281f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe2c83f640 1 -- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe2815e2a0 msgr2=0x7efe28156ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe2c83f640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe2815e2a0 0x7efe28156ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe2c83f640 1 -- 172.21.15.161:0/2831715549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe20009400 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe2c83f640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe280cd1c0 0x7efe28156960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efe1800c430 tx=0x7efe1800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe1dffb640 1 -- 172.21.15.161:0/2831715549 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe180052d0 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe1dffb640 1 -- 172.21.15.161:0/2831715549 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe18005450 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe1dffb640 1 -- 172.21.15.161:0/2831715549 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe18003830 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe28157a90 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.428+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe280c76b0 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.429+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe2804f550 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.429+0000 7efe1dffb640 1 -- 172.21.15.161:0/2831715549 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efe18003990 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.430+0000 7efe1dffb640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efe0c06b810 0x7efe0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.430+0000 7efe1dffb640 1 -- 172.21.15.161:0/2831715549 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7efe18089750 con 0x7efe280cd1c0 2024-03-23T14:35:52.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.430+0000 7efe2d040640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efe0c06b810 0x7efe0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.430+0000 7efe2d040640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efe0c06b810 0x7efe0c06dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efe24001560 tx=0x7efe24001a90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.434+0000 7efe1dffb640 1 -- 172.21.15.161:0/2831715549 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efe18054220 con 0x7efe280cd1c0 2024-03-23T14:35:52.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.605+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efe28158220 con 0x7efe280cd1c0 2024-03-23T14:35:52.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.605+0000 7efe1dffb640 1 -- 172.21.15.161:0/2831715549 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v20) ==== 104+0+11 (secure 0 0 0) 0x7efe28158220 con 0x7efe280cd1c0 2024-03-23T14:35:52.601 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:52.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efe0c06b810 msgr2=0x7efe0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efe0c06b810 0x7efe0c06dcd0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7efe24001560 tx=0x7efe24001a90 comp rx=0 tx=0).stop 2024-03-23T14:35:52.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe280cd1c0 msgr2=0x7efe28156960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe280cd1c0 0x7efe28156960 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7efe1800c430 tx=0x7efe1800c8f0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 shutdown_connections 2024-03-23T14:35:52.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efe0c06b810 0x7efe0c06dcd0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efe2815e2a0 0x7efe28156ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efe280cd1c0 0x7efe28156960 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 --2- 172.21.15.161:0/2831715549 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efe280cb2d0 0x7efe281f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 >> 172.21.15.161:0/2831715549 conn(0x7efe280b1e60 msgr2=0x7efe28066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:52.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 shutdown_connections 2024-03-23T14:35:52.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.607+0000 7efe2f2a4640 1 -- 172.21.15.161:0/2831715549 wait complete. 2024-03-23T14:35:52.614 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:52.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.721+0000 7f888926d640 1 Processor -- start 2024-03-23T14:35:52.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.721+0000 7f888926d640 1 -- start start 2024-03-23T14:35:52.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.722+0000 7f888926d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f888406cd20 0x7f888415e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.722+0000 7f888926d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8884070ee0 con 0x7f8884071370 2024-03-23T14:35:52.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.722+0000 7f888926d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88840b33b0 con 0x7f888415a2b0 2024-03-23T14:35:52.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.722+0000 7f888926d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f88840b3650 con 0x7f888406cd20 2024-03-23T14:35:52.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.722+0000 7f88827fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f888406cd20 0x7f888415e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.722+0000 7f88837fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8884071370 0x7f88840717e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37334/0 (socket says 172.21.15.161:37334) 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.722+0000 7f88837fe640 1 -- 172.21.15.161:0/756398680 learned_addr learned my addr 172.21.15.161:0/756398680 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.723+0000 7f8881ffb640 1 -- 172.21.15.161:0/756398680 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 511306969 0 0) 0x7f88840b33b0 con 0x7f888415a2b0 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.723+0000 7f88827fc640 1 -- 172.21.15.161:0/756398680 >> v1:172.21.15.183:6789/0 conn(0x7f888415a2b0 legacy=0x7f888415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.723+0000 7f88827fc640 1 -- 172.21.15.161:0/756398680 >> v1:172.21.15.161:6789/0 conn(0x7f8884071370 legacy=0x7f88840717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.723+0000 7f88827fc640 1 -- 172.21.15.161:0/756398680 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88840b3930 con 0x7f888406cd20 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.723+0000 7f88827fc640 1 --2- 172.21.15.161:0/756398680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f888415e690 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f88780097a0 tx=0x7f887802b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14ca5f9a916c0923 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f8881ffb640 1 -- 172.21.15.161:0/756398680 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878030710 con 0x7f888406cd20 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f8881ffb640 1 -- 172.21.15.161:0/756398680 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f887802e050 con 0x7f888406cd20 2024-03-23T14:35:52.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f8881ffb640 1 -- 172.21.15.161:0/756398680 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8878030a90 con 0x7f888406cd20 2024-03-23T14:35:52.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f888926d640 1 -- 172.21.15.161:0/756398680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 msgr2=0x7f888415e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f888926d640 1 --2- 172.21.15.161:0/756398680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f888415e690 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f88780097a0 tx=0x7f887802b790 comp rx=0 tx=0).stop 2024-03-23T14:35:52.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f888926d640 1 -- 172.21.15.161:0/756398680 shutdown_connections 2024-03-23T14:35:52.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f888926d640 1 --2- 172.21.15.161:0/756398680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f888415e690 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f888926d640 1 -- 172.21.15.161:0/756398680 >> 172.21.15.161:0/756398680 conn(0x7f888406b260 msgr2=0x7f888406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f888926d640 1 -- 172.21.15.161:0/756398680 shutdown_connections 2024-03-23T14:35:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.724+0000 7f888926d640 1 -- 172.21.15.161:0/756398680 wait complete. 2024-03-23T14:35:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f888926d640 1 Processor -- start 2024-03-23T14:35:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f888926d640 1 -- start start 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f888926d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f8884142dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f888926d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8884071370 0x7f888413d3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f888926d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f888415a2b0 0x7f8884158450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f888926d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8884143310 con 0x7f888406cd20 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f888926d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f88841438a0 con 0x7f888415a2b0 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f888926d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f888413d8e0 con 0x7f8884071370 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f8882ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f8884142dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f8882ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f8884142dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57150/0 (socket says 172.21.15.161:57150) 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f8882ffd640 1 -- 172.21.15.161:0/3765064139 learned_addr learned my addr 172.21.15.161:0/3765064139 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f88837fe640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f888415a2b0 0x7f8884158450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.726+0000 7f88827fc640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8884071370 0x7f888413d3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f8882ffd640 1 -- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8884071370 msgr2=0x7f888413d3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f8882ffd640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8884071370 0x7f888413d3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f8882ffd640 1 -- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f888415a2b0 msgr2=0x7f8884158450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f8882ffd640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f888415a2b0 0x7f8884158450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f8882ffd640 1 -- 172.21.15.161:0/3765064139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8878009400 con 0x7f888406cd20 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f8882ffd640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f8884142dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f887400c7f0 tx=0x7f887400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f886b7fe640 1 -- 172.21.15.161:0/3765064139 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874003da0 con 0x7f888406cd20 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f886b7fe640 1 -- 172.21.15.161:0/3765064139 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8874005070 con 0x7f888406cd20 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f886b7fe640 1 -- 172.21.15.161:0/3765064139 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8874005400 con 0x7f888406cd20 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88841435c0 con 0x7f888406cd20 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.727+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88841572d0 con 0x7f888406cd20 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.728+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8884068680 con 0x7f888406cd20 2024-03-23T14:35:52.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.728+0000 7f886b7fe640 1 -- 172.21.15.161:0/3765064139 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8874005560 con 0x7f888406cd20 2024-03-23T14:35:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.729+0000 7f886b7fe640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f887006b810 0x7f887006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.729+0000 7f886b7fe640 1 -- 172.21.15.161:0/3765064139 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8874089f30 con 0x7f888406cd20 2024-03-23T14:35:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.729+0000 7f88827fc640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f887006b810 0x7f887006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:52.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.729+0000 7f88827fc640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f887006b810 0x7f887006dcd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8878034000 tx=0x7f887802f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:52.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.733+0000 7f886b7fe640 1 -- 172.21.15.161:0/3765064139 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8874054a90 con 0x7f888406cd20 2024-03-23T14:35:52.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.908+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8884072740 con 0x7f888406cd20 2024-03-23T14:35:52.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.908+0000 7f886b7fe640 1 -- 172.21.15.161:0/3765064139 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f8874005810 con 0x7f888406cd20 2024-03-23T14:35:52.904 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:52.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f887006b810 msgr2=0x7f887006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f887006b810 0x7f887006dcd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8878034000 tx=0x7f887802f000 comp rx=0 tx=0).stop 2024-03-23T14:35:52.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 msgr2=0x7f8884142dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:52.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f8884142dd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f887400c7f0 tx=0x7f887400ccb0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 shutdown_connections 2024-03-23T14:35:52.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f887006b810 0x7f887006dcd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f888415a2b0 0x7f8884158450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8884071370 0x7f888413d3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 --2- 172.21.15.161:0/3765064139 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f888406cd20 0x7f8884142dd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:52.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 >> 172.21.15.161:0/3765064139 conn(0x7f888406b260 msgr2=0x7f88840b1c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:52.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 shutdown_connections 2024-03-23T14:35:52.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:52.910+0000 7f888926d640 1 -- 172.21.15.161:0/3765064139 wait complete. 2024-03-23T14:35:52.917 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:35:53.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.023+0000 7f467e433640 1 Processor -- start 2024-03-23T14:35:53.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.023+0000 7f467e433640 1 -- start start 2024-03-23T14:35:53.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.024+0000 7f467e433640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f46780d1470 0x7f46780d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.024+0000 7f467e433640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46780d0460 con 0x7f46780d2970 2024-03-23T14:35:53.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.024+0000 7f467e433640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46780d06b0 con 0x7f467815e400 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.024+0000 7f467e433640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f46780d1050 con 0x7f46780d1470 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.024+0000 7f467c9d0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f46780d2970 0x7f46780b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37336/0 (socket says 172.21.15.161:37336) 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.024+0000 7f467c9d0640 1 -- 172.21.15.161:0/3513262313 learned_addr learned my addr 172.21.15.161:0/3513262313 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.024+0000 7f46777fe640 1 --2- 172.21.15.161:0/3513262313 >> v2:172.21.15.183:3300/0 conn(0x7f46780d1470 0x7f46780d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.025+0000 7f4676ffd640 1 -- 172.21.15.161:0/3513262313 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2764579084 0 0) 0x7f46780d0460 con 0x7f46780d2970 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.025+0000 7f4676ffd640 1 -- 172.21.15.161:0/3513262313 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4660003520 con 0x7f46780d2970 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.025+0000 7f4676ffd640 1 -- 172.21.15.161:0/3513262313 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1649793940 0 0) 0x7f46780d06b0 con 0x7f467815e400 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.025+0000 7f4676ffd640 1 -- 172.21.15.161:0/3513262313 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f46780d0460 con 0x7f467815e400 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.025+0000 7f46777fe640 1 -- 172.21.15.161:0/3513262313 >> v1:172.21.15.183:6789/0 conn(0x7f467815e400 legacy=0x7f46780cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.025+0000 7f46777fe640 1 -- 172.21.15.161:0/3513262313 >> v1:172.21.15.161:6789/0 conn(0x7f46780d2970 legacy=0x7f46780b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.025+0000 7f46777fe640 1 -- 172.21.15.161:0/3513262313 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46780d0990 con 0x7f46780d1470 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f46777fe640 1 --2- 172.21.15.161:0/3513262313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46780d1470 0x7f46780d2430 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f466c0097a0 tx=0x7f466c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b97c9f6e63b8d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f4676ffd640 1 -- 172.21.15.161:0/3513262313 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c030610 con 0x7f46780d1470 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f4676ffd640 1 -- 172.21.15.161:0/3513262313 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f466c030770 con 0x7f46780d1470 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f4676ffd640 1 -- 172.21.15.161:0/3513262313 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c030b00 con 0x7f46780d1470 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f467e433640 1 -- 172.21.15.161:0/3513262313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46780d1470 msgr2=0x7f46780d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f467e433640 1 --2- 172.21.15.161:0/3513262313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46780d1470 0x7f46780d2430 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f466c0097a0 tx=0x7f466c02b960 comp rx=0 tx=0).stop 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f467e433640 1 -- 172.21.15.161:0/3513262313 shutdown_connections 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f467e433640 1 --2- 172.21.15.161:0/3513262313 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f46780d1470 0x7f46780d2430 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f467e433640 1 -- 172.21.15.161:0/3513262313 >> 172.21.15.161:0/3513262313 conn(0x7f467809d190 msgr2=0x7f46780d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f467e433640 1 -- 172.21.15.161:0/3513262313 shutdown_connections 2024-03-23T14:35:53.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.026+0000 7f467e433640 1 -- 172.21.15.161:0/3513262313 wait complete. 2024-03-23T14:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.027+0000 7f467e433640 1 Processor -- start 2024-03-23T14:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f467e433640 1 -- start start 2024-03-23T14:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f467e433640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46780d1470 0x7f467814acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f467e433640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46780d2970 0x7f46781f4060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f467e433640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467815e400 0x7f46781f7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f467e433640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f467814a2d0 con 0x7f46780d2970 2024-03-23T14:35:53.028 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f467e433640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f46781f5060 con 0x7f467815e400 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f467e433640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f467814b1f0 con 0x7f46780d1470 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f46777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46780d2970 0x7f46781f4060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f46777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46780d2970 0x7f46781f4060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57162/0 (socket says 172.21.15.161:57162) 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f46777fe640 1 -- 172.21.15.161:0/4038015552 learned_addr learned my addr 172.21.15.161:0/4038015552 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.028+0000 7f467c9d0640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467815e400 0x7f46781f7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f4677fff640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46780d1470 0x7f467814acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f46777fe640 1 -- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46780d1470 msgr2=0x7f467814acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f46777fe640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46780d1470 0x7f467814acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.029 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f46777fe640 1 -- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467815e400 msgr2=0x7f46781f7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f46777fe640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467815e400 0x7f46781f7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f46777fe640 1 -- 172.21.15.161:0/4038015552 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f466c009400 con 0x7f46780d2970 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f46777fe640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46780d2970 0x7f46781f4060 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f466c0098c0 tx=0x7f466c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f4674ff9640 1 -- 172.21.15.161:0/4038015552 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c03e070 con 0x7f46780d2970 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f4674ff9640 1 -- 172.21.15.161:0/4038015552 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f466c0397a0 con 0x7f46780d2970 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46781f4aa0 con 0x7f46780d2970 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f4674ff9640 1 -- 172.21.15.161:0/4038015552 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f466c039b70 con 0x7f46780d2970 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.029+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f467814a4b0 con 0x7f46780d2970 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.030+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46780a1e00 con 0x7f46780d2970 2024-03-23T14:35:53.030 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.030+0000 7f4674ff9640 1 -- 172.21.15.161:0/4038015552 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f466c039cd0 con 0x7f46780d2970 2024-03-23T14:35:53.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.031+0000 7f4674ff9640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f466406b810 0x7f466406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.031+0000 7f4674ff9640 1 -- 172.21.15.161:0/4038015552 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f466c0ac470 con 0x7f46780d2970 2024-03-23T14:35:53.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.031+0000 7f4677fff640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f466406b810 0x7f466406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.031+0000 7f4677fff640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f466406b810 0x7f466406dcd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4668007fd0 tx=0x7f4668009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.031 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.035+0000 7f4674ff9640 1 -- 172.21.15.161:0/4038015552 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f466c076e40 con 0x7f46780d2970 2024-03-23T14:35:53.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.219+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f46780b9c60 con 0x7f46780d2970 2024-03-23T14:35:53.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.219+0000 7f4674ff9640 1 -- 172.21.15.161:0/4038015552 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v21) ==== 80+0+0 (secure 0 0 0) 0x7f466c07a500 con 0x7f46780d2970 2024-03-23T14:35:53.215 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:35:53.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f466406b810 msgr2=0x7f466406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f466406b810 0x7f466406dcd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f4668007fd0 tx=0x7f4668009040 comp rx=0 tx=0).stop 2024-03-23T14:35:53.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46780d2970 msgr2=0x7f46781f4060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46780d2970 0x7f46781f4060 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f466c0098c0 tx=0x7f466c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 shutdown_connections 2024-03-23T14:35:53.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f466406b810 0x7f466406dcd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f467815e400 0x7f46781f7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f46780d2970 0x7f46781f4060 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 --2- 172.21.15.161:0/4038015552 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f46780d1470 0x7f467814acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 >> 172.21.15.161:0/4038015552 conn(0x7f467809d190 msgr2=0x7f46780b20c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:53.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 shutdown_connections 2024-03-23T14:35:53.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.221+0000 7f467e433640 1 -- 172.21.15.161:0/4038015552 wait complete. 2024-03-23T14:35:53.228 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi183.front.sepia.ceph.com... 2024-03-23T14:35:53.228 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:53.228 DEBUG:teuthology.orchestra.run.smithi183:> sudo fusermount -u /home/ubuntu/cephtest/mnt.3 2024-03-23T14:35:53.256 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:35:53.256 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:35:53.256 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:35:53.256 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T14:35:53.312 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T14:35:53.312 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:53.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.417+0000 7f4841af4640 1 Processor -- start 2024-03-23T14:35:53.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.417+0000 7f4841af4640 1 -- start start 2024-03-23T14:35:53.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f4841af4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f483c0724e0 0x7f483c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f4841af4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c0b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f4841af4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f483c15be70 con 0x7f483c072100 2024-03-23T14:35:53.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f4841af4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f483c06ef80 con 0x7f483c0724e0 2024-03-23T14:35:53.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f4841af4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f483c15c430 con 0x7f483c06ead0 2024-03-23T14:35:53.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483affd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f483c0724e0 0x7f483c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483bfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c0b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f483c072100 0x7f483c15b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33104/0 (socket says 172.21.15.161:33104) 2024-03-23T14:35:53.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483affd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f483c0724e0 0x7f483c06e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:54462/0 (socket says 172.21.15.161:54462) 2024-03-23T14:35:53.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483b7fe640 1 -- 172.21.15.161:0/1077268547 learned_addr learned my addr 172.21.15.161:0/1077268547 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:53.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483bfff640 1 -- 172.21.15.161:0/1077268547 >> v2:172.21.15.183:3300/0 conn(0x7f483c0724e0 msgr2=0x7f483c06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:53.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483bfff640 1 --2- 172.21.15.161:0/1077268547 >> v2:172.21.15.183:3300/0 conn(0x7f483c0724e0 0x7f483c06e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483bfff640 1 -- 172.21.15.161:0/1077268547 >> v1:172.21.15.183:6789/0 conn(0x7f483c072100 legacy=0x7f483c15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483bfff640 1 -- 172.21.15.161:0/1077268547 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f483c15c150 con 0x7f483c06ead0 2024-03-23T14:35:53.415 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.418+0000 7f483affd640 1 --2- 172.21.15.161:0/1077268547 >> v2:172.21.15.183:3300/0 conn(0x7f483c0724e0 0x7f483c06e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.419+0000 7f483bfff640 1 --2- 172.21.15.161:0/1077268547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c0b3470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4834009d90 tx=0x7f483402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d385cef09daf2c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.419+0000 7f483a7fc640 1 -- 172.21.15.161:0/1077268547 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f483402cc50 con 0x7f483c06ead0 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.419+0000 7f483a7fc640 1 -- 172.21.15.161:0/1077268547 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f483402cdb0 con 0x7f483c06ead0 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.419+0000 7f483a7fc640 1 -- 172.21.15.161:0/1077268547 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4834034660 con 0x7f483c06ead0 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.420+0000 7f4841af4640 1 -- 172.21.15.161:0/1077268547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 msgr2=0x7f483c0b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.420+0000 7f4841af4640 1 --2- 172.21.15.161:0/1077268547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c0b3470 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f4834009d90 tx=0x7f483402bea0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.420+0000 7f4841af4640 1 -- 172.21.15.161:0/1077268547 shutdown_connections 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.420+0000 7f4841af4640 1 --2- 172.21.15.161:0/1077268547 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c0b3470 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.420+0000 7f4841af4640 1 --2- 172.21.15.161:0/1077268547 >> v2:172.21.15.183:3300/0 conn(0x7f483c0724e0 0x7f483c06e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.420+0000 7f4841af4640 1 -- 172.21.15.161:0/1077268547 >> 172.21.15.161:0/1077268547 conn(0x7f483c0b17f0 msgr2=0x7f483c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.420+0000 7f4841af4640 1 -- 172.21.15.161:0/1077268547 shutdown_connections 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.420+0000 7f4841af4640 1 -- 172.21.15.161:0/1077268547 wait complete. 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.421+0000 7f4841af4640 1 Processor -- start 2024-03-23T14:35:53.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.421+0000 7f4841af4640 1 -- start start 2024-03-23T14:35:53.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f4841af4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c147c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f4841af4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f483c072100 0x7f483c15b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f4841af4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0724e0 0x7f483c14f970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f4841af4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f483c144140 con 0x7f483c072100 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f4841af4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f483c141df0 con 0x7f483c0724e0 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f4841af4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f483c144390 con 0x7f483c06ead0 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f483c072100 0x7f483c15b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483b7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c147c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f483c072100 0x7f483c15b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57176/0 (socket says 172.21.15.161:57176) 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483affd640 1 -- 172.21.15.161:0/2126990374 learned_addr learned my addr 172.21.15.161:0/2126990374 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483bfff640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0724e0 0x7f483c14f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483b7fe640 1 -- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0724e0 msgr2=0x7f483c14f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483b7fe640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0724e0 0x7f483c14f970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483b7fe640 1 -- 172.21.15.161:0/2126990374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f483c072100 msgr2=0x7f483c15b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483b7fe640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f483c072100 0x7f483c15b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483b7fe640 1 -- 172.21.15.161:0/2126990374 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4830009400 con 0x7f483c06ead0 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483bfff640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0724e0 0x7f483c14f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.422+0000 7f483affd640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f483c072100 0x7f483c15b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.423+0000 7f483b7fe640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c147c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f482c000f60 tx=0x7f482c00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.423+0000 7f4823fff640 1 -- 172.21.15.161:0/2126990374 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c005ce0 con 0x7f483c06ead0 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.423+0000 7f4823fff640 1 -- 172.21.15.161:0/2126990374 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f482c005e40 con 0x7f483c06ead0 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.423+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4834009a70 con 0x7f483c06ead0 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.423+0000 7f4823fff640 1 -- 172.21.15.161:0/2126990374 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f482c007310 con 0x7f483c06ead0 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.423+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f483c144950 con 0x7f483c06ead0 2024-03-23T14:35:53.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.424+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f483c0735e0 con 0x7f483c06ead0 2024-03-23T14:35:53.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.424+0000 7f4823fff640 1 -- 172.21.15.161:0/2126990374 <== mon.2 v2:172.21.15.183:3301/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f482c007470 con 0x7f483c06ead0 2024-03-23T14:35:53.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.424+0000 7f4823fff640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f482806b840 0x7f482806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.425+0000 7f4823fff640 1 -- 172.21.15.161:0/2126990374 <== mon.2 v2:172.21.15.183:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f482c089700 con 0x7f483c06ead0 2024-03-23T14:35:53.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.425+0000 7f483affd640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f482806b840 0x7f482806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.425+0000 7f483affd640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f482806b840 0x7f482806dd00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f483c154400 tx=0x7f4830009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.429+0000 7f4823fff640 1 -- 172.21.15.161:0/2126990374 <== mon.2 v2:172.21.15.183:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f482c054150 con 0x7f483c06ead0 2024-03-23T14:35:53.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.639+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f483c154c00 con 0x7f483c06ead0 2024-03-23T14:35:53.636 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.640+0000 7f4823fff640 1 -- 172.21.15.161:0/2126990374 <== mon.2 v2:172.21.15.183:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5181 (secure 0 0 0) 0x7f482c057810 con 0x7f483c06ead0 2024-03-23T14:35:53.636 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:53.636 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:35:13.112471+0000","modified":"2024-03-23T14:35:15.108804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-23T14:35:53.637 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 6 2024-03-23T14:35:53.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f482806b840 msgr2=0x7f482806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f482806b840 0x7f482806dd00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f483c154400 tx=0x7f4830009190 comp rx=0 tx=0).stop 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 msgr2=0x7f483c147c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c147c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f482c000f60 tx=0x7f482c00c800 comp rx=0 tx=0).stop 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 shutdown_connections 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f482806b840 0x7f482806dd00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f483c0724e0 0x7f483c14f970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f483c072100 0x7f483c15b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 --2- 172.21.15.161:0/2126990374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f483c06ead0 0x7f483c147c20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 >> 172.21.15.161:0/2126990374 conn(0x7f483c0b17f0 msgr2=0x7f483c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 shutdown_connections 2024-03-23T14:35:53.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.642+0000 7f4841af4640 1 -- 172.21.15.161:0/2126990374 wait complete. 2024-03-23T14:35:53.649 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:53.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.754+0000 7f447ba4e640 1 Processor -- start 2024-03-23T14:35:53.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.754+0000 7f447ba4e640 1 -- start start 2024-03-23T14:35:53.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f447ba4e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4474072100 0x7f447415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f447ba4e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44740724e0 0x7f447406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f447ba4e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f447415be70 con 0x7f447415b930 2024-03-23T14:35:53.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f447ba4e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f447415c430 con 0x7f44740724e0 2024-03-23T14:35:53.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f447ba4e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f447415c150 con 0x7f4474072100 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f4478fe9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44740724e0 0x7f447406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f4478fe9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f44740724e0 0x7f447406e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57190/0 (socket says 172.21.15.161:57190) 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f4478fe9640 1 -- 172.21.15.161:0/921303700 learned_addr learned my addr 172.21.15.161:0/921303700 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f4479feb640 1 --1- 172.21.15.161:0/921303700 >> v1:172.21.15.161:6789/0 conn(0x7f447415b930 0x7f447415e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37348/0 (socket says 172.21.15.161:37348) 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.755+0000 7f44797ea640 1 --2- 172.21.15.161:0/921303700 >> v2:172.21.15.183:3300/0 conn(0x7f4474072100 0x7f447415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.756+0000 7f4478fe9640 1 -- 172.21.15.161:0/921303700 >> v2:172.21.15.183:3300/0 conn(0x7f4474072100 msgr2=0x7f447415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.756+0000 7f4478fe9640 1 --2- 172.21.15.161:0/921303700 >> v2:172.21.15.183:3300/0 conn(0x7f4474072100 0x7f447415b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.756+0000 7f4478fe9640 1 -- 172.21.15.161:0/921303700 >> v1:172.21.15.161:6789/0 conn(0x7f447415b930 legacy=0x7f447415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.756+0000 7f4478fe9640 1 -- 172.21.15.161:0/921303700 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44740b2c70 con 0x7f44740724e0 2024-03-23T14:35:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.756+0000 7f446bfff640 1 -- 172.21.15.161:0/921303700 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3526435284 0 0) 0x7f447415be70 con 0x7f447415b930 2024-03-23T14:35:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.756+0000 7f44797ea640 1 --2- 172.21.15.161:0/921303700 >> v2:172.21.15.183:3300/0 conn(0x7f4474072100 0x7f447415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.756+0000 7f4478fe9640 1 --2- 172.21.15.161:0/921303700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44740724e0 0x7f447406e590 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f447406cac0 tx=0x7f446402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec0a58a78692e83 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.756+0000 7f446bfff640 1 -- 172.21.15.161:0/921303700 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464030860 con 0x7f44740724e0 2024-03-23T14:35:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f446bfff640 1 -- 172.21.15.161:0/921303700 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44640309c0 con 0x7f44740724e0 2024-03-23T14:35:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f446bfff640 1 -- 172.21.15.161:0/921303700 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4464030d50 con 0x7f44740724e0 2024-03-23T14:35:53.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f447ba4e640 1 -- 172.21.15.161:0/921303700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44740724e0 msgr2=0x7f447406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f447ba4e640 1 --2- 172.21.15.161:0/921303700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44740724e0 0x7f447406e590 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f447406cac0 tx=0x7f446402b9a0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f447ba4e640 1 -- 172.21.15.161:0/921303700 shutdown_connections 2024-03-23T14:35:53.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f447ba4e640 1 --2- 172.21.15.161:0/921303700 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f44740724e0 0x7f447406e590 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f447ba4e640 1 --2- 172.21.15.161:0/921303700 >> v2:172.21.15.183:3300/0 conn(0x7f4474072100 0x7f447415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f447ba4e640 1 -- 172.21.15.161:0/921303700 >> 172.21.15.161:0/921303700 conn(0x7f44740b17f0 msgr2=0x7f44740b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:53.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f447ba4e640 1 -- 172.21.15.161:0/921303700 shutdown_connections 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.757+0000 7f447ba4e640 1 -- 172.21.15.161:0/921303700 wait complete. 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.758+0000 7f447ba4e640 1 Processor -- start 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.758+0000 7f447ba4e640 1 -- start start 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f447ba4e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4474072100 0x7f447406e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f447ba4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44740724e0 0x7f4474150f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f447ba4e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f447415b930 0x7f4474144520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f447ba4e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f447414d970 con 0x7f4474072100 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f447ba4e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f447414b710 con 0x7f447415b930 2024-03-23T14:35:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f447ba4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f447414abd0 con 0x7f44740724e0 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f44797ea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4474072100 0x7f447406e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f44797ea640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4474072100 0x7f447406e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57192/0 (socket says 172.21.15.161:57192) 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f44797ea640 1 -- 172.21.15.161:0/1168970493 learned_addr learned my addr 172.21.15.161:0/1168970493 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f4478fe9640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44740724e0 0x7f4474150f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f4479feb640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f447415b930 0x7f4474144520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f44797ea640 1 -- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44740724e0 msgr2=0x7f4474150f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f44797ea640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44740724e0 0x7f4474150f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f44797ea640 1 -- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f447415b930 msgr2=0x7f4474144520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f44797ea640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f447415b930 0x7f4474144520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.759+0000 7f44797ea640 1 -- 172.21.15.161:0/1168970493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4470009a70 con 0x7f4474072100 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.760+0000 7f44797ea640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4474072100 0x7f447406e800 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f447000efc0 tx=0x7f447000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.760+0000 7f4469ffb640 1 -- 172.21.15.161:0/1168970493 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470014680 con 0x7f4474072100 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.760+0000 7f4469ffb640 1 -- 172.21.15.161:0/1168970493 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f447000f040 con 0x7f4474072100 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.760+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44640094d0 con 0x7f4474072100 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.760+0000 7f4469ffb640 1 -- 172.21.15.161:0/1168970493 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4470014a00 con 0x7f4474072100 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.760+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f447414b150 con 0x7f4474072100 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.761+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44740735e0 con 0x7f4474072100 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.761+0000 7f4469ffb640 1 -- 172.21.15.161:0/1168970493 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4470014b60 con 0x7f4474072100 2024-03-23T14:35:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.762+0000 7f4469ffb640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f445806b810 0x7f445806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:53.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.762+0000 7f4469ffb640 1 -- 172.21.15.161:0/1168970493 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5825+0+0 (secure 0 0 0) 0x7f447008b140 con 0x7f4474072100 2024-03-23T14:35:53.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.762+0000 7f4478fe9640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f445806b810 0x7f445806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:53.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.762+0000 7f4478fe9640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f445806b810 0x7f445806dcd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4464036000 tx=0x7f4464008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:53.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.766+0000 7f4469ffb640 1 -- 172.21.15.161:0/1168970493 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4470055c20 con 0x7f4474072100 2024-03-23T14:35:53.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.951+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f447404f550 con 0x7f4474072100 2024-03-23T14:35:53.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.952+0000 7f4469ffb640 1 -- 172.21.15.161:0/1168970493 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16831 (secure 0 0 0) 0x7f44700592e0 con 0x7f4474072100 2024-03-23T14:35:53.948 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:53.948 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":22,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:53.858945+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.24","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:53.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f445806b810 msgr2=0x7f445806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f445806b810 0x7f445806dcd0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f4464036000 tx=0x7f4464008030 comp rx=0 tx=0).stop 2024-03-23T14:35:53.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4474072100 msgr2=0x7f447406e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:53.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4474072100 0x7f447406e800 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f447000efc0 tx=0x7f447000c490 comp rx=0 tx=0).stop 2024-03-23T14:35:53.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 shutdown_connections 2024-03-23T14:35:53.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f445806b810 0x7f445806dcd0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f447415b930 0x7f4474144520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f44740724e0 0x7f4474150f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 --2- 172.21.15.161:0/1168970493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4474072100 0x7f447406e800 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:53.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 >> 172.21.15.161:0/1168970493 conn(0x7f44740b17f0 msgr2=0x7f447414fb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:53.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 shutdown_connections 2024-03-23T14:35:53.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:53.954+0000 7f447ba4e640 1 -- 172.21.15.161:0/1168970493 wait complete. 2024-03-23T14:35:53.961 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:35:11.179507+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:35:12.073445+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:35:53.961 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:35:54.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.070+0000 7f9b76a15640 1 Processor -- start 2024-03-23T14:35:54.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.070+0000 7f9b76a15640 1 -- start start 2024-03-23T14:35:54.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b76a15640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9b700b2aa0 0x7f9b700cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b76a15640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9b700d01b0 0x7f9b7006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b76a15640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b700d0600 con 0x7f9b701552f0 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b76a15640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9b7015f480 con 0x7f9b700d01b0 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b76a15640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f9b701556d0 con 0x7f9b700b2aa0 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b74fb2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9b700d01b0 0x7f9b7006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b6ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9b701552f0 0x7f9b700b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37352/0 (socket says 172.21.15.161:37352) 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b74fb2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9b700d01b0 0x7f9b7006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57196/0 (socket says 172.21.15.161:57196) 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b6ffff640 1 -- 172.21.15.161:0/1771407166 learned_addr learned my addr 172.21.15.161:0/1771407166 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b6f7fe640 1 --2- 172.21.15.161:0/1771407166 >> v2:172.21.15.183:3300/0 conn(0x7f9b700b2aa0 0x7f9b700cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b74fb2640 1 -- 172.21.15.161:0/1771407166 >> v2:172.21.15.183:3300/0 conn(0x7f9b700b2aa0 msgr2=0x7f9b700cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b74fb2640 1 --2- 172.21.15.161:0/1771407166 >> v2:172.21.15.183:3300/0 conn(0x7f9b700b2aa0 0x7f9b700cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b74fb2640 1 -- 172.21.15.161:0/1771407166 >> v1:172.21.15.161:6789/0 conn(0x7f9b701552f0 legacy=0x7f9b700b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.071+0000 7f9b74fb2640 1 -- 172.21.15.161:0/1771407166 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b7015e870 con 0x7f9b700d01b0 2024-03-23T14:35:54.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.072+0000 7f9b74fb2640 1 --2- 172.21.15.161:0/1771407166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b7006ded0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9b7006ce20 tx=0x7f9b6802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6da243c1b9955322 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.072+0000 7f9b6effd640 1 -- 172.21.15.161:0/1771407166 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6802cc80 con 0x7f9b700d01b0 2024-03-23T14:35:54.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.072+0000 7f9b6effd640 1 -- 172.21.15.161:0/1771407166 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6802cde0 con 0x7f9b700d01b0 2024-03-23T14:35:54.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.072+0000 7f9b6effd640 1 -- 172.21.15.161:0/1771407166 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b68034660 con 0x7f9b700d01b0 2024-03-23T14:35:54.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.073+0000 7f9b76a15640 1 -- 172.21.15.161:0/1771407166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 msgr2=0x7f9b7006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.073+0000 7f9b76a15640 1 --2- 172.21.15.161:0/1771407166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b7006ded0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f9b7006ce20 tx=0x7f9b6802bed0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.073+0000 7f9b76a15640 1 -- 172.21.15.161:0/1771407166 shutdown_connections 2024-03-23T14:35:54.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.073+0000 7f9b76a15640 1 --2- 172.21.15.161:0/1771407166 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b7006ded0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.073+0000 7f9b76a15640 1 --2- 172.21.15.161:0/1771407166 >> v2:172.21.15.183:3300/0 conn(0x7f9b700b2aa0 0x7f9b700cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.073+0000 7f9b76a15640 1 -- 172.21.15.161:0/1771407166 >> 172.21.15.161:0/1771407166 conn(0x7f9b7006b260 msgr2=0x7f9b7006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:54.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.073+0000 7f9b76a15640 1 -- 172.21.15.161:0/1771407166 shutdown_connections 2024-03-23T14:35:54.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.073+0000 7f9b76a15640 1 -- 172.21.15.161:0/1771407166 wait complete. 2024-03-23T14:35:54.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.074+0000 7f9b76a15640 1 Processor -- start 2024-03-23T14:35:54.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.074+0000 7f9b76a15640 1 -- start start 2024-03-23T14:35:54.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b76a15640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b700b2aa0 0x7f9b7014b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b76a15640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b700b1cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b76a15640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b701552f0 0x7f9b70153d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b76a15640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9b700b0320 con 0x7f9b700d01b0 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b76a15640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9b700af710 con 0x7f9b700b2aa0 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b76a15640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9b700af9f0 con 0x7f9b701552f0 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b700b1cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b700b1cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57198/0 (socket says 172.21.15.161:57198) 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6f7fe640 1 -- 172.21.15.161:0/294644940 learned_addr learned my addr 172.21.15.161:0/294644940 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6ffff640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b700b2aa0 0x7f9b7014b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b74fb2640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b701552f0 0x7f9b70153d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6f7fe640 1 -- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b701552f0 msgr2=0x7f9b70153d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6f7fe640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b701552f0 0x7f9b70153d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6f7fe640 1 -- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b700b2aa0 msgr2=0x7f9b7014b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6f7fe640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b700b2aa0 0x7f9b7014b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6f7fe640 1 -- 172.21.15.161:0/294644940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b68009a70 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.075+0000 7f9b6ffff640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b700b2aa0 0x7f9b7014b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.076+0000 7f9b6f7fe640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b700b1cd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400e9b0 tx=0x7f9b6400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.076+0000 7f9b6cff9640 1 -- 172.21.15.161:0/294644940 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b64009670 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.076+0000 7f9b6cff9640 1 -- 172.21.15.161:0/294644940 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b6400c9a0 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.076+0000 7f9b6cff9640 1 -- 172.21.15.161:0/294644940 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b6400cd50 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.076+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b700afcd0 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.076+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b7014b9e0 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.077+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b700b04d0 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.077+0000 7f9b6cff9640 1 -- 172.21.15.161:0/294644940 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9b64013470 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.077+0000 7f9b6cff9640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b5c06b810 0x7f9b5c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.078+0000 7f9b6cff9640 1 -- 172.21.15.161:0/294644940 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f9b6408a350 con 0x7f9b700d01b0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.078+0000 7f9b6ffff640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b5c06b810 0x7f9b5c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.078+0000 7f9b6ffff640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b5c06b810 0x7f9b5c06dcd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b60007fd0 tx=0x7f9b60007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.082+0000 7f9b6cff9640 1 -- 172.21.15.161:0/294644940 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9b700b04d0 con 0x7f9b700d01b0 2024-03-23T14:35:54.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.253+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9b7004f4e0 con 0x7f9b700d01b0 2024-03-23T14:35:54.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.253+0000 7f9b6cff9640 1 -- 172.21.15.161:0/294644940 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v22) ==== 93+0+10 (secure 0 0 0) 0x7f9b64054e30 con 0x7f9b700d01b0 2024-03-23T14:35:54.249 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:35:54.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b5c06b810 msgr2=0x7f9b5c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b5c06b810 0x7f9b5c06dcd0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9b60007fd0 tx=0x7f9b60007e80 comp rx=0 tx=0).stop 2024-03-23T14:35:54.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 msgr2=0x7f9b700b1cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b700b1cd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9b6400e9b0 tx=0x7f9b6400ee70 comp rx=0 tx=0).stop 2024-03-23T14:35:54.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 shutdown_connections 2024-03-23T14:35:54.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b5c06b810 0x7f9b5c06dcd0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b701552f0 0x7f9b70153d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b700d01b0 0x7f9b700b1cd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 --2- 172.21.15.161:0/294644940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b700b2aa0 0x7f9b7014b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 >> 172.21.15.161:0/294644940 conn(0x7f9b7006b260 msgr2=0x7f9b70159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:54.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 shutdown_connections 2024-03-23T14:35:54.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.255+0000 7f9b76a15640 1 -- 172.21.15.161:0/294644940 wait complete. 2024-03-23T14:35:54.262 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:35:54.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.370+0000 7f18301f4640 1 Processor -- start 2024-03-23T14:35:54.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.370+0000 7f18301f4640 1 -- start start 2024-03-23T14:35:54.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f18301f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b4010 0x7f18280cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f18301f4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18280b43f0 0x7f18280d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f18301f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f182815c1f0 con 0x7f182815b480 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f18301f4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f182806ef20 con 0x7f18280b43f0 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f18301f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f182815f730 con 0x7f18280b4010 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f182d78f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18280b43f0 0x7f18280d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f182d78f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18280b43f0 0x7f18280d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57214/0 (socket says 172.21.15.161:57214) 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f182d78f640 1 -- 172.21.15.161:0/3380426241 learned_addr learned my addr 172.21.15.161:0/3380426241 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f182df90640 1 --2- 172.21.15.161:0/3380426241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b4010 0x7f18280cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f182d78f640 1 -- 172.21.15.161:0/3380426241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b4010 msgr2=0x7f18280cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f182d78f640 1 --2- 172.21.15.161:0/3380426241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b4010 0x7f18280cfee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f182d78f640 1 -- 172.21.15.161:0/3380426241 >> v1:172.21.15.183:6789/0 conn(0x7f182815b480 legacy=0x7f182815b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.371+0000 7f182d78f640 1 -- 172.21.15.161:0/3380426241 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f182815e970 con 0x7f18280b43f0 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.372+0000 7f182d78f640 1 --2- 172.21.15.161:0/3380426241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b43f0 0x7f18280d0420 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1818009870 tx=0x7f181802b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b052cb2cd5e98410 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.372+0000 7f182cf8e640 1 -- 172.21.15.161:0/3380426241 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818031640 con 0x7f18280b43f0 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.372+0000 7f182cf8e640 1 -- 172.21.15.161:0/3380426241 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18180317a0 con 0x7f18280b43f0 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.372+0000 7f182cf8e640 1 -- 172.21.15.161:0/3380426241 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1818031b30 con 0x7f18280b43f0 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.373+0000 7f18301f4640 1 -- 172.21.15.161:0/3380426241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b43f0 msgr2=0x7f18280d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.373+0000 7f18301f4640 1 --2- 172.21.15.161:0/3380426241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b43f0 0x7f18280d0420 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f1818009870 tx=0x7f181802b8f0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.373+0000 7f18301f4640 1 -- 172.21.15.161:0/3380426241 shutdown_connections 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.373+0000 7f18301f4640 1 --2- 172.21.15.161:0/3380426241 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b43f0 0x7f18280d0420 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.373+0000 7f18301f4640 1 --2- 172.21.15.161:0/3380426241 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b4010 0x7f18280cfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.373+0000 7f18301f4640 1 -- 172.21.15.161:0/3380426241 >> 172.21.15.161:0/3380426241 conn(0x7f182806b4a0 msgr2=0x7f182806b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:54.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.373+0000 7f18301f4640 1 -- 172.21.15.161:0/3380426241 shutdown_connections 2024-03-23T14:35:54.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.373+0000 7f18301f4640 1 -- 172.21.15.161:0/3380426241 wait complete. 2024-03-23T14:35:54.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.374+0000 7f18301f4640 1 Processor -- start 2024-03-23T14:35:54.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.374+0000 7f18301f4640 1 -- start start 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f18301f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b4010 0x7f182813f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f18301f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b43f0 0x7f18280b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f18301f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182815b480 0x7f18280b0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f18301f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f182814a820 con 0x7f18280b4010 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f18301f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1828153b20 con 0x7f182815b480 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f18301f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1828155a20 con 0x7f18280b43f0 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b4010 0x7f182813f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b4010 0x7f182813f470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57216/0 (socket says 172.21.15.161:57216) 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 -- 172.21.15.161:0/3550370743 learned_addr learned my addr 172.21.15.161:0/3550370743 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182e791640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182815b480 0x7f18280b0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182d78f640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b43f0 0x7f18280b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 -- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b43f0 msgr2=0x7f18280b0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b43f0 0x7f18280b0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 -- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182815b480 msgr2=0x7f18280b0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182815b480 0x7f18280b0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 -- 172.21.15.161:0/3550370743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18180094d0 con 0x7f18280b4010 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.375+0000 7f182df90640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b4010 0x7f182813f470 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1824009db0 tx=0x7f182400c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.376+0000 7f181e7fc640 1 -- 172.21.15.161:0/3550370743 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824014480 con 0x7f18280b4010 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.376+0000 7f181e7fc640 1 -- 172.21.15.161:0/3550370743 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f182400f040 con 0x7f18280b4010 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.376+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1828155d00 con 0x7f18280b4010 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.376+0000 7f181e7fc640 1 -- 172.21.15.161:0/3550370743 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1824014800 con 0x7f18280b4010 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.376+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18281562c0 con 0x7f18280b4010 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.377+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f182806de90 con 0x7f18280b4010 2024-03-23T14:35:54.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.377+0000 7f181e7fc640 1 -- 172.21.15.161:0/3550370743 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1824014960 con 0x7f18280b4010 2024-03-23T14:35:54.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.377+0000 7f181e7fc640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f180c06b810 0x7f180c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.378+0000 7f181e7fc640 1 -- 172.21.15.161:0/3550370743 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f182408acd0 con 0x7f18280b4010 2024-03-23T14:35:54.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.378+0000 7f182d78f640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f180c06b810 0x7f180c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.378+0000 7f182d78f640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f180c06b810 0x7f180c06dcd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1818036000 tx=0x7f181803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.382+0000 7f181e7fc640 1 -- 172.21.15.161:0/3550370743 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1824055720 con 0x7f18280b4010 2024-03-23T14:35:54.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.553+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1828073570 con 0x7f18280b4010 2024-03-23T14:35:54.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.553+0000 7f181e7fc640 1 -- 172.21.15.161:0/3550370743 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v22) ==== 104+0+11 (secure 0 0 0) 0x7f1824058de0 con 0x7f18280b4010 2024-03-23T14:35:54.549 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:35:54.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f180c06b810 msgr2=0x7f180c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f180c06b810 0x7f180c06dcd0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f1818036000 tx=0x7f181803c040 comp rx=0 tx=0).stop 2024-03-23T14:35:54.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b4010 msgr2=0x7f182813f470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b4010 0x7f182813f470 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f1824009db0 tx=0x7f182400c490 comp rx=0 tx=0).stop 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 shutdown_connections 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f180c06b810 0x7f180c06dcd0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f182815b480 0x7f18280b0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18280b43f0 0x7f18280b0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 --2- 172.21.15.161:0/3550370743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18280b4010 0x7f182813f470 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 >> 172.21.15.161:0/3550370743 conn(0x7f182806b4a0 msgr2=0x7f18280b2870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 shutdown_connections 2024-03-23T14:35:54.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.555+0000 7f18301f4640 1 -- 172.21.15.161:0/3550370743 wait complete. 2024-03-23T14:35:54.562 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:35:54.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.670+0000 7f783d73e640 1 Processor -- start 2024-03-23T14:35:54.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.670+0000 7f783d73e640 1 -- start start 2024-03-23T14:35:54.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f783d73e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7838154a80 0x7f78380cf230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f783d73e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 0x7f78380b3570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f783d73e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f783806e4f0 con 0x7f783815df10 2024-03-23T14:35:54.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f783d73e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f783815ed80 con 0x7f7838154a80 2024-03-23T14:35:54.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f783d73e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7838154e60 con 0x7f783806dd90 2024-03-23T14:35:54.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f7836ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7838154a80 0x7f78380cf230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f7836ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7838154a80 0x7f78380cf230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57230/0 (socket says 172.21.15.161:57230) 2024-03-23T14:35:54.668 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f7836ffd640 1 -- 172.21.15.161:0/1084895514 learned_addr learned my addr 172.21.15.161:0/1084895514 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f78367fc640 1 --2- 172.21.15.161:0/1084895514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 0x7f78380b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f7836ffd640 1 -- 172.21.15.161:0/1084895514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 msgr2=0x7f78380b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f7836ffd640 1 --2- 172.21.15.161:0/1084895514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 0x7f78380b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f7836ffd640 1 -- 172.21.15.161:0/1084895514 >> v1:172.21.15.183:6789/0 conn(0x7f783815df10 legacy=0x7f7838070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.671+0000 7f7836ffd640 1 -- 172.21.15.161:0/1084895514 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78380b2ad0 con 0x7f7838154a80 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.672+0000 7f7836ffd640 1 --2- 172.21.15.161:0/1084895514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cf230 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7828009e50 tx=0x7f782802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be1177e8616b72dc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.672+0000 7f7835ffb640 1 -- 172.21.15.161:0/1084895514 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782802ca70 con 0x7f7838154a80 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.672+0000 7f7835ffb640 1 -- 172.21.15.161:0/1084895514 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782802cbd0 con 0x7f7838154a80 2024-03-23T14:35:54.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.672+0000 7f7835ffb640 1 -- 172.21.15.161:0/1084895514 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78280343f0 con 0x7f7838154a80 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.673+0000 7f783d73e640 1 -- 172.21.15.161:0/1084895514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 msgr2=0x7f78380cf230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.673+0000 7f783d73e640 1 --2- 172.21.15.161:0/1084895514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cf230 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f7828009e50 tx=0x7f782802bed0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.673+0000 7f783d73e640 1 -- 172.21.15.161:0/1084895514 shutdown_connections 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.673+0000 7f783d73e640 1 --2- 172.21.15.161:0/1084895514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 0x7f78380b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.673+0000 7f783d73e640 1 --2- 172.21.15.161:0/1084895514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cf230 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.673+0000 7f783d73e640 1 -- 172.21.15.161:0/1084895514 >> 172.21.15.161:0/1084895514 conn(0x7f783806b260 msgr2=0x7f783806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.673+0000 7f783d73e640 1 -- 172.21.15.161:0/1084895514 shutdown_connections 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.673+0000 7f783d73e640 1 -- 172.21.15.161:0/1084895514 wait complete. 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.674+0000 7f783d73e640 1 Processor -- start 2024-03-23T14:35:54.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.674+0000 7f783d73e640 1 -- start start 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f783d73e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 0x7f78381392a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f783d73e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f783d73e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783815df10 0x7f78380ccaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f783d73e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7838150260 con 0x7f7838154a80 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f783d73e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7838150540 con 0x7f783815df10 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f783d73e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f78380af6b0 con 0x7f783806dd90 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f7836ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 0x7f78381392a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57232/0 (socket says 172.21.15.161:57232) 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 -- 172.21.15.161:0/452687774 learned_addr learned my addr 172.21.15.161:0/452687774 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78377fe640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783815df10 0x7f78380ccaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 -- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 msgr2=0x7f78381392a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 0x7f78381392a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 -- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783815df10 msgr2=0x7f78380ccaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783815df10 0x7f78380ccaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 -- 172.21.15.161:0/452687774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7828009b30 con 0x7f7838154a80 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78377fe640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783815df10 0x7f78380ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:54.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.675+0000 7f78367fc640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cc5b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f782c00e870 tx=0x7f782c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.676+0000 7f781f7fe640 1 -- 172.21.15.161:0/452687774 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c0094f0 con 0x7f7838154a80 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.676+0000 7f781f7fe640 1 -- 172.21.15.161:0/452687774 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f782c00c940 con 0x7f7838154a80 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.676+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78380af990 con 0x7f7838154a80 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.676+0000 7f781f7fe640 1 -- 172.21.15.161:0/452687774 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782c00ccf0 con 0x7f7838154a80 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.676+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78381397e0 con 0x7f7838154a80 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.677+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f783804f4e0 con 0x7f7838154a80 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.677+0000 7f781f7fe640 1 -- 172.21.15.161:0/452687774 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f782c013470 con 0x7f7838154a80 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.677+0000 7f781f7fe640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f782406b840 0x7f782406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.678+0000 7f781f7fe640 1 -- 172.21.15.161:0/452687774 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f782c08a220 con 0x7f7838154a80 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.678+0000 7f7836ffd640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f782406b840 0x7f782406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.678+0000 7f7836ffd640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f782406b840 0x7f782406dd00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7828035000 tx=0x7f7828030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.682+0000 7f781f7fe640 1 -- 172.21.15.161:0/452687774 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f782c054c70 con 0x7f7838154a80 2024-03-23T14:35:54.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.851+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f78380b3610 con 0x7f7838154a80 2024-03-23T14:35:54.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.852+0000 7f781f7fe640 1 -- 172.21.15.161:0/452687774 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v22) ==== 100+0+11 (secure 0 0 0) 0x7f782c058330 con 0x7f7838154a80 2024-03-23T14:35:54.848 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f782406b840 msgr2=0x7f782406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f782406b840 0x7f782406dd00 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f7828035000 tx=0x7f7828030000 comp rx=0 tx=0).stop 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 msgr2=0x7f78380cc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cc5b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f782c00e870 tx=0x7f782c00ed30 comp rx=0 tx=0).stop 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 shutdown_connections 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f782406b840 0x7f782406dd00 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f783815df10 0x7f78380ccaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7838154a80 0x7f78380cc5b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 --2- 172.21.15.161:0/452687774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f783806dd90 0x7f78381392a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 >> 172.21.15.161:0/452687774 conn(0x7f783806b260 msgr2=0x7f783815e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:54.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 shutdown_connections 2024-03-23T14:35:54.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.854+0000 7f783d73e640 1 -- 172.21.15.161:0/452687774 wait complete. 2024-03-23T14:35:54.862 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:54.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.967+0000 7ffaeea55640 1 Processor -- start 2024-03-23T14:35:54.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.967+0000 7ffaeea55640 1 -- start start 2024-03-23T14:35:54.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.967+0000 7ffaeea55640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffae80724e0 0x7ffae806e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.967+0000 7ffaeea55640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae815be70 con 0x7ffae8072100 2024-03-23T14:35:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.967+0000 7ffaeea55640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffae815c150 con 0x7ffae806ead0 2024-03-23T14:35:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.967+0000 7ffaeea55640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffae806ef80 con 0x7ffae80724e0 2024-03-23T14:35:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.968+0000 7ffae77fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffae80724e0 0x7ffae806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.968+0000 7ffae77fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffae80724e0 0x7ffae806e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57244/0 (socket says 172.21.15.161:57244) 2024-03-23T14:35:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.968+0000 7ffae77fe640 1 -- 172.21.15.161:0/3364882516 learned_addr learned my addr 172.21.15.161:0/3364882516 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.968+0000 7ffae7fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ffae8072100 0x7ffae815b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37368/0 (socket says 172.21.15.161:37368) 2024-03-23T14:35:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.968+0000 7ffae77fe640 1 -- 172.21.15.161:0/3364882516 >> v1:172.21.15.183:6789/0 conn(0x7ffae806ead0 legacy=0x7ffae80b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.968+0000 7ffae77fe640 1 -- 172.21.15.161:0/3364882516 >> v1:172.21.15.161:6789/0 conn(0x7ffae8072100 legacy=0x7ffae815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.968+0000 7ffae77fe640 1 -- 172.21.15.161:0/3364882516 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffae80b2c70 con 0x7ffae80724e0 2024-03-23T14:35:54.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.968+0000 7ffae6ffd640 1 -- 172.21.15.161:0/3364882516 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3956661666 0 0) 0x7ffae815be70 con 0x7ffae8072100 2024-03-23T14:35:54.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffae77fe640 1 --2- 172.21.15.161:0/3364882516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae80724e0 0x7ffae806e590 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00bb10 tx=0x7ffadc02e1f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b7513f4191b334 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffae6ffd640 1 -- 172.21.15.161:0/3364882516 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc036440 con 0x7ffae80724e0 2024-03-23T14:35:54.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffae6ffd640 1 -- 172.21.15.161:0/3364882516 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffadc0365a0 con 0x7ffae80724e0 2024-03-23T14:35:54.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffae6ffd640 1 -- 172.21.15.161:0/3364882516 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc036930 con 0x7ffae80724e0 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffaeea55640 1 -- 172.21.15.161:0/3364882516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae80724e0 msgr2=0x7ffae806e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffaeea55640 1 --2- 172.21.15.161:0/3364882516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae80724e0 0x7ffae806e590 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00bb10 tx=0x7ffadc02e1f0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffaeea55640 1 -- 172.21.15.161:0/3364882516 shutdown_connections 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffaeea55640 1 --2- 172.21.15.161:0/3364882516 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae80724e0 0x7ffae806e590 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffaeea55640 1 -- 172.21.15.161:0/3364882516 >> 172.21.15.161:0/3364882516 conn(0x7ffae80b17f0 msgr2=0x7ffae80b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffaeea55640 1 -- 172.21.15.161:0/3364882516 shutdown_connections 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.969+0000 7ffaeea55640 1 -- 172.21.15.161:0/3364882516 wait complete. 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffaeea55640 1 Processor -- start 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffaeea55640 1 -- start start 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffaeea55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffae806ead0 0x7ffae813f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffaeea55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae8072100 0x7ffae81e4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffaeea55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffae80724e0 0x7ffae814f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffaeea55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81e34f0 con 0x7ffae8072100 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffaeea55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffae81e3740 con 0x7ffae80724e0 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffaeea55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ffae81e3a20 con 0x7ffae806ead0 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.971+0000 7ffae77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae8072100 0x7ffae81e4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae8072100 0x7ffae81e4500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57254/0 (socket says 172.21.15.161:57254) 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae77fe640 1 -- 172.21.15.161:0/727901403 learned_addr learned my addr 172.21.15.161:0/727901403 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae7fff640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffae806ead0 0x7ffae813f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffaecff2640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffae80724e0 0x7ffae814f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae77fe640 1 -- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffae806ead0 msgr2=0x7ffae813f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae77fe640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffae806ead0 0x7ffae813f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae77fe640 1 -- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffae80724e0 msgr2=0x7ffae814f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae77fe640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffae80724e0 0x7ffae814f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae77fe640 1 -- 172.21.15.161:0/727901403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffadc00b770 con 0x7ffae8072100 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae77fe640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae8072100 0x7ffae81e4500 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00b2c0 tx=0x7ffadc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae4ff9640 1 -- 172.21.15.161:0/727901403 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc034b00 con 0x7ffae8072100 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae4ff9640 1 -- 172.21.15.161:0/727901403 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffadc034c80 con 0x7ffae8072100 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffae81e3d00 con 0x7ffae8072100 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffae4ff9640 1 -- 172.21.15.161:0/727901403 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffadc041440 con 0x7ffae8072100 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.972+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffae814f2a0 con 0x7ffae8072100 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.973+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffae81e4220 con 0x7ffae8072100 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.974+0000 7ffae4ff9640 1 -- 172.21.15.161:0/727901403 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffadc03d040 con 0x7ffae8072100 2024-03-23T14:35:54.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.974+0000 7ffae4ff9640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad406b810 0x7ffad406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:54.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.974+0000 7ffae4ff9640 1 -- 172.21.15.161:0/727901403 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7ffadc0af000 con 0x7ffae8072100 2024-03-23T14:35:54.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.974+0000 7ffae7fff640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad406b810 0x7ffad406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:54.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.975+0000 7ffae7fff640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad406b810 0x7ffad406dcd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffad8007f70 tx=0x7ffad8007f00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:54.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:54.978+0000 7ffae4ff9640 1 -- 172.21.15.161:0/727901403 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffae81e4220 con 0x7ffae8072100 2024-03-23T14:35:55.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.185+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffae81e4220 con 0x7ffae8072100 2024-03-23T14:35:55.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.185+0000 7ffae4ff9640 1 -- 172.21.15.161:0/727901403 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5181 (secure 0 0 0) 0x7ffae81e4220 con 0x7ffae8072100 2024-03-23T14:35:55.182 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:55.182 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:35:13.112471+0000","modified":"2024-03-23T14:35:15.108804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-23T14:35:55.182 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 6 2024-03-23T14:35:55.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad406b810 msgr2=0x7ffad406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad406b810 0x7ffad406dcd0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ffad8007f70 tx=0x7ffad8007f00 comp rx=0 tx=0).stop 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae8072100 msgr2=0x7ffae81e4500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae8072100 0x7ffae81e4500 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7ffadc00b2c0 tx=0x7ffadc0046c0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 shutdown_connections 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffad406b810 0x7ffad406dcd0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffae80724e0 0x7ffae814f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffae8072100 0x7ffae81e4500 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 --2- 172.21.15.161:0/727901403 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffae806ead0 0x7ffae813f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 >> 172.21.15.161:0/727901403 conn(0x7ffae80b17f0 msgr2=0x7ffae80b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 shutdown_connections 2024-03-23T14:35:55.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.188+0000 7ffaeea55640 1 -- 172.21.15.161:0/727901403 wait complete. 2024-03-23T14:35:55.197 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:55.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.312+0000 7f694c15c640 1 Processor -- start 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.312+0000 7f694c15c640 1 -- start start 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f694c15c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944072100 0x7f694415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f694c15c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f694406eca0 0x7f69440b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f694c15c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f694415be70 con 0x7f69440724e0 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f694c15c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f694415c430 con 0x7f694406eca0 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f694c15c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f694415c150 con 0x7f6944072100 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f694a6f9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f694406eca0 0x7f69440b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f694a6f9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f694406eca0 0x7f69440b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57266/0 (socket says 172.21.15.161:57266) 2024-03-23T14:35:55.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f694a6f9640 1 -- 172.21.15.161:0/4057317414 learned_addr learned my addr 172.21.15.161:0/4057317414 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.313+0000 7f6949ef8640 1 --2- 172.21.15.161:0/4057317414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944072100 0x7f694415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f694a6f9640 1 -- 172.21.15.161:0/4057317414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944072100 msgr2=0x7f694415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f694a6f9640 1 --2- 172.21.15.161:0/4057317414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944072100 0x7f694415b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f694a6f9640 1 -- 172.21.15.161:0/4057317414 >> v1:172.21.15.183:6789/0 conn(0x7f69440724e0 legacy=0x7f694406e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f6948ef6640 1 -- 172.21.15.161:0/4057317414 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3447152649 0 0) 0x7f694415be70 con 0x7f69440724e0 2024-03-23T14:35:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f694a6f9640 1 -- 172.21.15.161:0/4057317414 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69440b2c70 con 0x7f694406eca0 2024-03-23T14:35:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f694a6f9640 1 --2- 172.21.15.161:0/4057317414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f694406eca0 0x7f69440b3470 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f693c009870 tx=0x7f693c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34d4a51995f47f5c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f6948ef6640 1 -- 172.21.15.161:0/4057317414 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c030840 con 0x7f694406eca0 2024-03-23T14:35:55.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f6948ef6640 1 -- 172.21.15.161:0/4057317414 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693c0309a0 con 0x7f694406eca0 2024-03-23T14:35:55.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.314+0000 7f6948ef6640 1 -- 172.21.15.161:0/4057317414 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c030d30 con 0x7f694406eca0 2024-03-23T14:35:55.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.315+0000 7f694c15c640 1 -- 172.21.15.161:0/4057317414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f694406eca0 msgr2=0x7f69440b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.315+0000 7f694c15c640 1 --2- 172.21.15.161:0/4057317414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f694406eca0 0x7f69440b3470 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f693c009870 tx=0x7f693c02b980 comp rx=0 tx=0).stop 2024-03-23T14:35:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.315+0000 7f694c15c640 1 -- 172.21.15.161:0/4057317414 shutdown_connections 2024-03-23T14:35:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.315+0000 7f694c15c640 1 --2- 172.21.15.161:0/4057317414 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f694406eca0 0x7f69440b3470 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.315+0000 7f694c15c640 1 --2- 172.21.15.161:0/4057317414 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6944072100 0x7f694415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.315+0000 7f694c15c640 1 -- 172.21.15.161:0/4057317414 >> 172.21.15.161:0/4057317414 conn(0x7f69440b17f0 msgr2=0x7f69440b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.315+0000 7f694c15c640 1 -- 172.21.15.161:0/4057317414 shutdown_connections 2024-03-23T14:35:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.315+0000 7f694c15c640 1 -- 172.21.15.161:0/4057317414 wait complete. 2024-03-23T14:35:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.316+0000 7f694c15c640 1 Processor -- start 2024-03-23T14:35:55.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.316+0000 7f694c15c640 1 -- start start 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f694c15c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f694406eca0 0x7f6944149750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f694c15c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6944072100 0x7f6944159460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f694c15c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69440724e0 0x7f694406d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f694c15c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f694415a850 con 0x7f6944072100 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f694c15c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f69441599a0 con 0x7f69440724e0 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f694c15c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f6944159c40 con 0x7f694406eca0 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f69496f7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6944072100 0x7f6944159460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f69496f7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6944072100 0x7f6944159460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57282/0 (socket says 172.21.15.161:57282) 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f69496f7640 1 -- 172.21.15.161:0/3358131340 learned_addr learned my addr 172.21.15.161:0/3358131340 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f694a6f9640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69440724e0 0x7f694406d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f6949ef8640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f694406eca0 0x7f6944149750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f69496f7640 1 -- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f694406eca0 msgr2=0x7f6944149750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f69496f7640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f694406eca0 0x7f6944149750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f69496f7640 1 -- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69440724e0 msgr2=0x7f694406d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f69496f7640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69440724e0 0x7f694406d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f69496f7640 1 -- 172.21.15.161:0/3358131340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f693c0094d0 con 0x7f6944072100 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f694a6f9640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69440724e0 0x7f694406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.317+0000 7f6949ef8640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f694406eca0 0x7f6944149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.318+0000 7f69496f7640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6944072100 0x7f6944159460 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f693400c410 tx=0x7f693400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.318+0000 7f693a7fc640 1 -- 172.21.15.161:0/3358131340 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69340035f0 con 0x7f6944072100 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.318+0000 7f693a7fc640 1 -- 172.21.15.161:0/3358131340 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6934003770 con 0x7f6944072100 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.318+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6944159f20 con 0x7f6944072100 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.318+0000 7f693a7fc640 1 -- 172.21.15.161:0/3358131340 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6934003b20 con 0x7f6944072100 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.318+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f694415a4e0 con 0x7f6944072100 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.319+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f694415aa00 con 0x7f6944072100 2024-03-23T14:35:55.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.319+0000 7f693a7fc640 1 -- 172.21.15.161:0/3358131340 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6934003c80 con 0x7f6944072100 2024-03-23T14:35:55.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.319+0000 7f693a7fc640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f692806b840 0x7f692806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.320+0000 7f693a7fc640 1 -- 172.21.15.161:0/3358131340 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f69340897a0 con 0x7f6944072100 2024-03-23T14:35:55.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.320+0000 7f6949ef8640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f692806b840 0x7f692806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.320+0000 7f6949ef8640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f692806b840 0x7f692806dd00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f694000e740 tx=0x7f694000b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.324+0000 7f693a7fc640 1 -- 172.21.15.161:0/3358131340 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f694415aa00 con 0x7f6944072100 2024-03-23T14:35:55.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.497+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f69440735e0 con 0x7f6944072100 2024-03-23T14:35:55.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.498+0000 7f693a7fc640 1 -- 172.21.15.161:0/3358131340 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16831 (secure 0 0 0) 0x7f69340541f0 con 0x7f6944072100 2024-03-23T14:35:55.494 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:55.494 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":22,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:53.858945+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.24","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:55.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f692806b840 msgr2=0x7f692806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f692806b840 0x7f692806dd00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f694000e740 tx=0x7f694000b040 comp rx=0 tx=0).stop 2024-03-23T14:35:55.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6944072100 msgr2=0x7f6944159460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6944072100 0x7f6944159460 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f693400c410 tx=0x7f693400c8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 shutdown_connections 2024-03-23T14:35:55.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f692806b840 0x7f692806dd00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f69440724e0 0x7f694406d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6944072100 0x7f6944159460 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 --2- 172.21.15.161:0/3358131340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f694406eca0 0x7f6944149750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 >> 172.21.15.161:0/3358131340 conn(0x7f69440b17f0 msgr2=0x7f694406e590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:55.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 shutdown_connections 2024-03-23T14:35:55.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.500+0000 7f694c15c640 1 -- 172.21.15.161:0/3358131340 wait complete. 2024-03-23T14:35:55.508 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T14:35:55.508 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.615+0000 7fa572f6b640 1 Processor -- start 2024-03-23T14:35:55.613 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.615+0000 7fa572f6b640 1 -- start start 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa572f6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c06eb30 0x7fa56c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa572f6b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa56c0b4260 0x7fa56c15b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa572f6b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa56c070b90 con 0x7fa56c0b3610 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa572f6b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa56c15ab90 con 0x7fa56c0b4260 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa572f6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa56c0713a0 con 0x7fa56c06eb30 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa56bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa56c0b4260 0x7fa56c15b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa56bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa56c0b4260 0x7fa56c15b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57298/0 (socket says 172.21.15.161:57298) 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa56bfff640 1 -- 172.21.15.161:0/2071994856 learned_addr learned my addr 172.21.15.161:0/2071994856 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa571508640 1 --1- 172.21.15.161:0/2071994856 >> v1:172.21.15.161:6789/0 conn(0x7fa56c0b3610 0x7fa56c06d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37378/0 (socket says 172.21.15.161:37378) 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.616+0000 7fa570d07640 1 --2- 172.21.15.161:0/2071994856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c06eb30 0x7fa56c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.617+0000 7fa56bfff640 1 -- 172.21.15.161:0/2071994856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c06eb30 msgr2=0x7fa56c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.617+0000 7fa56bfff640 1 --2- 172.21.15.161:0/2071994856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c06eb30 0x7fa56c0b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.617+0000 7fa56bfff640 1 -- 172.21.15.161:0/2071994856 >> v1:172.21.15.161:6789/0 conn(0x7fa56c0b3610 legacy=0x7fa56c06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.617+0000 7fa56bfff640 1 -- 172.21.15.161:0/2071994856 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa56c15b0d0 con 0x7fa56c0b4260 2024-03-23T14:35:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.617+0000 7fa56b7fe640 1 -- 172.21.15.161:0/2071994856 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2884694069 0 0) 0x7fa56c070b90 con 0x7fa56c0b3610 2024-03-23T14:35:55.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.617+0000 7fa56bfff640 1 --2- 172.21.15.161:0/2071994856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c0b4260 0x7fa56c15b560 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa560009870 tx=0x7fa56002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c919d32259b90b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa56b7fe640 1 -- 172.21.15.161:0/2071994856 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560030840 con 0x7fa56c0b4260 2024-03-23T14:35:55.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa56b7fe640 1 -- 172.21.15.161:0/2071994856 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5600309a0 con 0x7fa56c0b4260 2024-03-23T14:35:55.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa56b7fe640 1 -- 172.21.15.161:0/2071994856 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa560030d30 con 0x7fa56c0b4260 2024-03-23T14:35:55.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa572f6b640 1 -- 172.21.15.161:0/2071994856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c0b4260 msgr2=0x7fa56c15b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa572f6b640 1 --2- 172.21.15.161:0/2071994856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c0b4260 0x7fa56c15b560 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa560009870 tx=0x7fa56002b980 comp rx=0 tx=0).stop 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa572f6b640 1 -- 172.21.15.161:0/2071994856 shutdown_connections 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa572f6b640 1 --2- 172.21.15.161:0/2071994856 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c0b4260 0x7fa56c15b560 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa572f6b640 1 --2- 172.21.15.161:0/2071994856 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c06eb30 0x7fa56c0b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa572f6b640 1 -- 172.21.15.161:0/2071994856 >> 172.21.15.161:0/2071994856 conn(0x7fa56c0b17f0 msgr2=0x7fa56c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa572f6b640 1 -- 172.21.15.161:0/2071994856 shutdown_connections 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.618+0000 7fa572f6b640 1 -- 172.21.15.161:0/2071994856 wait complete. 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.619+0000 7fa572f6b640 1 Processor -- start 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.619+0000 7fa572f6b640 1 -- start start 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa572f6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c06eb30 0x7fa56c14de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa572f6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c0b3610 0x7fa56c147580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa572f6b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa56c0b4260 0x7fa56c0cea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa572f6b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c150d30 con 0x7fa56c06eb30 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa572f6b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa56c151010 con 0x7fa56c0b4260 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa572f6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa56c1529e0 con 0x7fa56c0b3610 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa570d07640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c06eb30 0x7fa56c14de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa570d07640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c06eb30 0x7fa56c14de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57308/0 (socket says 172.21.15.161:57308) 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa570d07640 1 -- 172.21.15.161:0/902483537 learned_addr learned my addr 172.21.15.161:0/902483537 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa571508640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa56c0b4260 0x7fa56c0cea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.620+0000 7fa56bfff640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c0b3610 0x7fa56c147580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa570d07640 1 -- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c0b3610 msgr2=0x7fa56c147580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa570d07640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c0b3610 0x7fa56c147580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa570d07640 1 -- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa56c0b4260 msgr2=0x7fa56c0cea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa570d07640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa56c0b4260 0x7fa56c0cea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa570d07640 1 -- 172.21.15.161:0/902483537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5600094d0 con 0x7fa56c06eb30 2024-03-23T14:35:55.620 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa571508640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa56c0b4260 0x7fa56c0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:35:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa570d07640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c06eb30 0x7fa56c14de10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00e9b0 tx=0x7fa55c00ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa5697fa640 1 -- 172.21.15.161:0/902483537 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c009670 con 0x7fa56c06eb30 2024-03-23T14:35:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa5697fa640 1 -- 172.21.15.161:0/902483537 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa55c00c9a0 con 0x7fa56c06eb30 2024-03-23T14:35:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa56c152cc0 con 0x7fa56c06eb30 2024-03-23T14:35:55.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa5697fa640 1 -- 172.21.15.161:0/902483537 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa55c00cd50 con 0x7fa56c06eb30 2024-03-23T14:35:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.621+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa56c147ac0 con 0x7fa56c06eb30 2024-03-23T14:35:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.622+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa56c152840 con 0x7fa56c06eb30 2024-03-23T14:35:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.622+0000 7fa5697fa640 1 -- 172.21.15.161:0/902483537 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa55c013470 con 0x7fa56c06eb30 2024-03-23T14:35:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.623+0000 7fa5697fa640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa55806b810 0x7fa55806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.623+0000 7fa5697fa640 1 -- 172.21.15.161:0/902483537 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fa55c08a3f0 con 0x7fa56c06eb30 2024-03-23T14:35:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.623+0000 7fa56bfff640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa55806b810 0x7fa55806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.623+0000 7fa56bfff640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa55806b810 0x7fa55806dcd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa560036000 tx=0x7fa560008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.627+0000 7fa5697fa640 1 -- 172.21.15.161:0/902483537 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa56c152840 con 0x7fa56c06eb30 2024-03-23T14:35:55.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.832+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa56c152840 con 0x7fa56c06eb30 2024-03-23T14:35:55.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.833+0000 7fa5697fa640 1 -- 172.21.15.161:0/902483537 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5181 (secure 0 0 0) 0x7fa56c152840 con 0x7fa56c06eb30 2024-03-23T14:35:55.829 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:55.829 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:35:13.112471+0000","modified":"2024-03-23T14:35:15.108804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-23T14:35:55.829 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 6 2024-03-23T14:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa55806b810 msgr2=0x7fa55806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa55806b810 0x7fa55806dcd0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fa560036000 tx=0x7fa560008030 comp rx=0 tx=0).stop 2024-03-23T14:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c06eb30 msgr2=0x7fa56c14de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c06eb30 0x7fa56c14de10 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fa55c00e9b0 tx=0x7fa55c00ee70 comp rx=0 tx=0).stop 2024-03-23T14:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 shutdown_connections 2024-03-23T14:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa55806b810 0x7fa55806dcd0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa56c0b4260 0x7fa56c0cea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa56c0b3610 0x7fa56c147580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 --2- 172.21.15.161:0/902483537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa56c06eb30 0x7fa56c14de10 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 >> 172.21.15.161:0/902483537 conn(0x7fa56c0b17f0 msgr2=0x7fa56c15cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 shutdown_connections 2024-03-23T14:35:55.832 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.835+0000 7fa572f6b640 1 -- 172.21.15.161:0/902483537 wait complete. 2024-03-23T14:35:55.842 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:35:55.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.948+0000 7ffb0439a640 1 Processor -- start 2024-03-23T14:35:55.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.948+0000 7ffb0439a640 1 -- start start 2024-03-23T14:35:55.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb0439a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffafc0b4260 0x7ffafc15b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb0439a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffafc070b90 con 0x7ffafc0b3610 2024-03-23T14:35:55.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb0439a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffafc15b0d0 con 0x7ffafc06eb30 2024-03-23T14:35:55.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb0439a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffafc15ab90 con 0x7ffafc0b4260 2024-03-23T14:35:55.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb01935640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffafc0b4260 0x7ffafc15b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb02937640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ffafc0b3610 0x7ffafc06d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37386/0 (socket says 172.21.15.161:37386) 2024-03-23T14:35:55.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb02937640 1 -- 172.21.15.161:0/3651232750 learned_addr learned my addr 172.21.15.161:0/3651232750 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:55.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb01935640 1 -- 172.21.15.161:0/3651232750 >> v1:172.21.15.183:6789/0 conn(0x7ffafc06eb30 legacy=0x7ffafc0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb01935640 1 -- 172.21.15.161:0/3651232750 >> v1:172.21.15.161:6789/0 conn(0x7ffafc0b3610 legacy=0x7ffafc06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb01935640 1 -- 172.21.15.161:0/3651232750 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc15ada0 con 0x7ffafc0b4260 2024-03-23T14:35:55.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.949+0000 7ffb01134640 1 -- 172.21.15.161:0/3651232750 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742448951 0 0) 0x7ffafc070b90 con 0x7ffafc0b3610 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.950+0000 7ffb01935640 1 --2- 172.21.15.161:0/3651232750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc0b4260 0x7ffafc15b560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0097a0 tx=0x7ffaec02b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba495cff63ce6ca1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.950+0000 7ffb01134640 1 -- 172.21.15.161:0/3651232750 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec030610 con 0x7ffafc0b4260 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.950+0000 7ffb01134640 1 -- 172.21.15.161:0/3651232750 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaec030770 con 0x7ffafc0b4260 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.950+0000 7ffb01134640 1 -- 172.21.15.161:0/3651232750 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaec030b00 con 0x7ffafc0b4260 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.951+0000 7ffb0439a640 1 -- 172.21.15.161:0/3651232750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc0b4260 msgr2=0x7ffafc15b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.951+0000 7ffb0439a640 1 --2- 172.21.15.161:0/3651232750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc0b4260 0x7ffafc15b560 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7ffaec0097a0 tx=0x7ffaec02b960 comp rx=0 tx=0).stop 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.951+0000 7ffb0439a640 1 -- 172.21.15.161:0/3651232750 shutdown_connections 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.951+0000 7ffb0439a640 1 --2- 172.21.15.161:0/3651232750 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc0b4260 0x7ffafc15b560 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.951+0000 7ffb0439a640 1 -- 172.21.15.161:0/3651232750 >> 172.21.15.161:0/3651232750 conn(0x7ffafc0b17f0 msgr2=0x7ffafc0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:55.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.951+0000 7ffb0439a640 1 -- 172.21.15.161:0/3651232750 shutdown_connections 2024-03-23T14:35:55.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.951+0000 7ffb0439a640 1 -- 172.21.15.161:0/3651232750 wait complete. 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.952+0000 7ffb0439a640 1 Processor -- start 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.952+0000 7ffb0439a640 1 -- start start 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb0439a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc06eb30 0x7ffafc12e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb0439a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffafc0b3610 0x7ffafc147f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb0439a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffafc0b4260 0x7ffafc1442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb0439a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc12eae0 con 0x7ffafc06eb30 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb0439a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc143210 con 0x7ffafc0b3610 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb0439a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ffafc1434f0 con 0x7ffafc0b4260 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc06eb30 0x7ffafc12e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc06eb30 0x7ffafc12e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57334/0 (socket says 172.21.15.161:57334) 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 -- 172.21.15.161:0/297914780 learned_addr learned my addr 172.21.15.161:0/297914780 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb01935640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffafc0b3610 0x7ffafc147f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02937640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffafc0b4260 0x7ffafc1442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 -- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffafc0b4260 msgr2=0x7ffafc1442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffafc0b4260 0x7ffafc1442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 -- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffafc0b3610 msgr2=0x7ffafc147f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffafc0b3610 0x7ffafc147f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 -- 172.21.15.161:0/297914780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaec009400 con 0x7ffafc06eb30 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb02136640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc06eb30 0x7ffafc12e5a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800c450 tx=0x7ffaf800c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffaf2ffd640 1 -- 172.21.15.161:0/297914780 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf80052d0 con 0x7ffafc06eb30 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffaf2ffd640 1 -- 172.21.15.161:0/297914780 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffaf8005430 con 0x7ffafc06eb30 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffafc1437d0 con 0x7ffafc06eb30 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffaf2ffd640 1 -- 172.21.15.161:0/297914780 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf8012870 con 0x7ffafc06eb30 2024-03-23T14:35:55.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.953+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffafc143d90 con 0x7ffafc06eb30 2024-03-23T14:35:55.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.954+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffafc143f80 con 0x7ffafc06eb30 2024-03-23T14:35:55.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.955+0000 7ffaf2ffd640 1 -- 172.21.15.161:0/297914780 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffaf80129d0 con 0x7ffafc06eb30 2024-03-23T14:35:55.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.955+0000 7ffaf2ffd640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffae006b810 0x7ffae006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:55.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.955+0000 7ffaf2ffd640 1 -- 172.21.15.161:0/297914780 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7ffaf8089c40 con 0x7ffafc06eb30 2024-03-23T14:35:55.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.955+0000 7ffb01935640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffae006b810 0x7ffae006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:55.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.956+0000 7ffb01935640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffae006b810 0x7ffae006dcd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffaec036000 tx=0x7ffaec008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:55.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:55.959+0000 7ffaf2ffd640 1 -- 172.21.15.161:0/297914780 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffaf8054690 con 0x7ffafc06eb30 2024-03-23T14:35:56.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.168+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffafc04f550 con 0x7ffafc06eb30 2024-03-23T14:35:56.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.168+0000 7ffaf2ffd640 1 -- 172.21.15.161:0/297914780 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+5181 (secure 0 0 0) 0x7ffaf8057d50 con 0x7ffafc06eb30 2024-03-23T14:35:56.165 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:56.165 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":6,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4275,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/35034401","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":35034401},{"type":"v1","addr":"172.21.15.161:6835","nonce":35034401}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4278,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/1560557568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":1560557568},{"type":"v1","addr":"172.21.15.161:6834","nonce":1560557568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4279,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2795806378","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2795806378},{"type":"v1","addr":"172.21.15.183:6836","nonce":2795806378}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":6,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:35:13.112471+0000","modified":"2024-03-23T14:35:15.108804+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":4282},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4282":{"gid":4282,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6837/1031097819","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":1031097819},{"type":"v1","addr":"172.21.15.183:6837","nonce":1031097819}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":4282,"qdb_cluster":[4282]},"id":1}]} 2024-03-23T14:35:56.165 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 6 2024-03-23T14:35:56.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffae006b810 msgr2=0x7ffae006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffae006b810 0x7ffae006dcd0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7ffaec036000 tx=0x7ffaec008030 comp rx=0 tx=0).stop 2024-03-23T14:35:56.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc06eb30 msgr2=0x7ffafc12e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc06eb30 0x7ffafc12e5a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7ffaf800c450 tx=0x7ffaf800c910 comp rx=0 tx=0).stop 2024-03-23T14:35:56.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 shutdown_connections 2024-03-23T14:35:56.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffae006b810 0x7ffae006dcd0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffafc0b4260 0x7ffafc1442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffafc0b3610 0x7ffafc147f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 --2- 172.21.15.161:0/297914780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffafc06eb30 0x7ffafc12e5a0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 >> 172.21.15.161:0/297914780 conn(0x7ffafc0b17f0 msgr2=0x7ffafc15cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:56.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 shutdown_connections 2024-03-23T14:35:56.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.171+0000 7ffb0439a640 1 -- 172.21.15.161:0/297914780 wait complete. 2024-03-23T14:35:56.178 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:35:56.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.283+0000 7f2d22d5e640 1 Processor -- start 2024-03-23T14:35:56.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.283+0000 7f2d22d5e640 1 -- start start 2024-03-23T14:35:56.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.284+0000 7f2d22d5e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2d1c0d2b40 0x7f2d1c0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.284+0000 7f2d22d5e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d1c0d0460 con 0x7f2d1c0d1470 2024-03-23T14:35:56.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.284+0000 7f2d22d5e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d1c0d06b0 con 0x7f2d1c15e400 2024-03-23T14:35:56.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.284+0000 7f2d22d5e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f2d1c0d1050 con 0x7f2d1c0d2b40 2024-03-23T14:35:56.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.284+0000 7f2d1bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2d1c0d1470 0x7f2d1c0d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37394/0 (socket says 172.21.15.161:37394) 2024-03-23T14:35:56.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.284+0000 7f2d1bfff640 1 -- 172.21.15.161:0/1981010472 learned_addr learned my addr 172.21.15.161:0/1981010472 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.284+0000 7f2d212fb640 1 --2- 172.21.15.161:0/1981010472 >> v2:172.21.15.183:3300/0 conn(0x7f2d1c0d2b40 0x7f2d1c0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.285+0000 7f2d1b7fe640 1 -- 172.21.15.161:0/1981010472 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3737607534 0 0) 0x7f2d1c0d0460 con 0x7f2d1c0d1470 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.285+0000 7f2d1b7fe640 1 -- 172.21.15.161:0/1981010472 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d04003520 con 0x7f2d1c0d1470 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.285+0000 7f2d1b7fe640 1 -- 172.21.15.161:0/1981010472 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1959609339 0 0) 0x7f2d1c0d06b0 con 0x7f2d1c15e400 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.285+0000 7f2d1b7fe640 1 -- 172.21.15.161:0/1981010472 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2d1c0d0460 con 0x7f2d1c15e400 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.285+0000 7f2d212fb640 1 -- 172.21.15.161:0/1981010472 >> v1:172.21.15.183:6789/0 conn(0x7f2d1c15e400 legacy=0x7f2d1c0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.285+0000 7f2d212fb640 1 -- 172.21.15.161:0/1981010472 >> v1:172.21.15.161:6789/0 conn(0x7f2d1c0d1470 legacy=0x7f2d1c0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.285+0000 7f2d212fb640 1 -- 172.21.15.161:0/1981010472 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d1c0d0990 con 0x7f2d1c0d2b40 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.285+0000 7f2d212fb640 1 --2- 172.21.15.161:0/1981010472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c0d2b40 0x7f2d1c0b0a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009d90 tx=0x7f2d1402bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc744d5dfd3845ab server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d1b7fe640 1 -- 172.21.15.161:0/1981010472 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d1402cd00 con 0x7f2d1c0d2b40 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d1b7fe640 1 -- 172.21.15.161:0/1981010472 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d1402ce60 con 0x7f2d1c0d2b40 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d1b7fe640 1 -- 172.21.15.161:0/1981010472 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d14034660 con 0x7f2d1c0d2b40 2024-03-23T14:35:56.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1981010472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c0d2b40 msgr2=0x7f2d1c0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d22d5e640 1 --2- 172.21.15.161:0/1981010472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c0d2b40 0x7f2d1c0b0a80 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2d14009d90 tx=0x7f2d1402bf50 comp rx=0 tx=0).stop 2024-03-23T14:35:56.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1981010472 shutdown_connections 2024-03-23T14:35:56.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d22d5e640 1 --2- 172.21.15.161:0/1981010472 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c0d2b40 0x7f2d1c0b0a80 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1981010472 >> 172.21.15.161:0/1981010472 conn(0x7f2d1c09d190 msgr2=0x7f2d1c0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:56.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1981010472 shutdown_connections 2024-03-23T14:35:56.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.286+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1981010472 wait complete. 2024-03-23T14:35:56.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.287+0000 7f2d22d5e640 1 Processor -- start 2024-03-23T14:35:56.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.287+0000 7f2d22d5e640 1 -- start start 2024-03-23T14:35:56.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d22d5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d1c0d1470 0x7f2d1c15b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d22d5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d1c0d2b40 0x7f2d1c0cb900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d22d5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c15e400 0x7f2d1c0cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d22d5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c0cd020 con 0x7f2d1c0d1470 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d22d5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2d1c0b6ca0 con 0x7f2d1c15e400 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d22d5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2d1c162a60 con 0x7f2d1c0d2b40 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d20afa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d1c0d1470 0x7f2d1c15b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d20afa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d1c0d1470 0x7f2d1c15b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57342/0 (socket says 172.21.15.161:57342) 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d20afa640 1 -- 172.21.15.161:0/1359818904 learned_addr learned my addr 172.21.15.161:0/1359818904 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d212fb640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c15e400 0x7f2d1c0cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.288+0000 7f2d1bfff640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d1c0d2b40 0x7f2d1c0cb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d20afa640 1 -- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d1c0d2b40 msgr2=0x7f2d1c0cb900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d20afa640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d1c0d2b40 0x7f2d1c0cb900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d20afa640 1 -- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c15e400 msgr2=0x7f2d1c0cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d20afa640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c15e400 0x7f2d1c0cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d20afa640 1 -- 172.21.15.161:0/1359818904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d14009a70 con 0x7f2d1c0d1470 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d20afa640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d1c0d1470 0x7f2d1c15b210 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00c500 tx=0x7f2d0c00c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d197fa640 1 -- 172.21.15.161:0/1359818904 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c0052d0 con 0x7f2d1c0d1470 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d197fa640 1 -- 172.21.15.161:0/1359818904 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d0c005450 con 0x7f2d1c0d1470 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d197fa640 1 -- 172.21.15.161:0/1359818904 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0c012870 con 0x7f2d1c0d1470 2024-03-23T14:35:56.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d1c0cc4a0 con 0x7f2d1c0d1470 2024-03-23T14:35:56.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.289+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d1c0cca60 con 0x7f2d1c0d1470 2024-03-23T14:35:56.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.290+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d1c0a1e00 con 0x7f2d1c0d1470 2024-03-23T14:35:56.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.290+0000 7f2d197fa640 1 -- 172.21.15.161:0/1359818904 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2d0c0129d0 con 0x7f2d1c0d1470 2024-03-23T14:35:56.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.291+0000 7f2d197fa640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2d0806b810 0x7f2d0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.291+0000 7f2d197fa640 1 -- 172.21.15.161:0/1359818904 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f2d0c089d40 con 0x7f2d1c0d1470 2024-03-23T14:35:56.290 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.291+0000 7f2d1bfff640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2d0806b810 0x7f2d0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.291+0000 7f2d1bfff640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2d0806b810 0x7f2d0806dcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0b4e90 tx=0x7f2d10007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.295+0000 7f2d197fa640 1 -- 172.21.15.161:0/1359818904 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2d0c0547a0 con 0x7f2d1c0d1470 2024-03-23T14:35:56.462 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.466+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2d1c04f550 con 0x7f2d1c0d1470 2024-03-23T14:35:56.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.467+0000 7f2d197fa640 1 -- 172.21.15.161:0/1359818904 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v22) ==== 74+0+16831 (secure 0 0 0) 0x7f2d0c057e60 con 0x7f2d1c0d1470 2024-03-23T14:35:56.464 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:35:56.464 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":22,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:35:53.858945+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:35:11.179507+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-23T14:35:12.073445+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.11","mappings":[{"from":7,"to":2}]},{"pgid":"3.17","mappings":[{"from":5,"to":1}]},{"pgid":"3.24","mappings":[{"from":3,"to":2}]},{"pgid":"3.35","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:35:56.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2d0806b810 msgr2=0x7f2d0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2d0806b810 0x7f2d0806dcd0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c0b4e90 tx=0x7f2d10007d30 comp rx=0 tx=0).stop 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d1c0d1470 msgr2=0x7f2d1c15b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d1c0d1470 0x7f2d1c15b210 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c00c500 tx=0x7f2d0c00c9c0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 shutdown_connections 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2d0806b810 0x7f2d0806dcd0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d1c15e400 0x7f2d1c0cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d1c0d2b40 0x7f2d1c0cb900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 --2- 172.21.15.161:0/1359818904 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d1c0d1470 0x7f2d1c15b210 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 >> 172.21.15.161:0/1359818904 conn(0x7f2d1c09d190 msgr2=0x7f2d1c15c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 shutdown_connections 2024-03-23T14:35:56.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.469+0000 7f2d22d5e640 1 -- 172.21.15.161:0/1359818904 wait complete. 2024-03-23T14:35:56.477 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T14:35:56.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.583+0000 7feab57f6640 1 Processor -- start 2024-03-23T14:35:56.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.583+0000 7feab57f6640 1 -- start start 2024-03-23T14:35:56.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feab57f6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feab015e400 0x7feab00cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feab57f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00d2b40 0x7feab00b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feab57f6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feab00d0460 con 0x7feab00cf9d0 2024-03-23T14:35:56.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feab57f6640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7feab00d1050 con 0x7feab015e400 2024-03-23T14:35:56.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feab57f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feab00d06b0 con 0x7feab00d2b40 2024-03-23T14:35:56.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaaeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feab015e400 0x7feab00cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaae7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7feab00cf9d0 0x7feab00d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37396/0 (socket says 172.21.15.161:37396) 2024-03-23T14:35:56.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaae7fc640 1 -- 172.21.15.161:0/3758183381 learned_addr learned my addr 172.21.15.161:0/3758183381 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaaf7fe640 1 --2- 172.21.15.161:0/3758183381 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00d2b40 0x7feab00b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaaeffd640 1 -- 172.21.15.161:0/3758183381 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00d2b40 msgr2=0x7feab00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaaeffd640 1 --2- 172.21.15.161:0/3758183381 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00d2b40 0x7feab00b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaaeffd640 1 -- 172.21.15.161:0/3758183381 >> v1:172.21.15.161:6789/0 conn(0x7feab00cf9d0 legacy=0x7feab00d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaaeffd640 1 -- 172.21.15.161:0/3758183381 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feab00d0990 con 0x7feab015e400 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.584+0000 7feaadffb640 1 -- 172.21.15.161:0/3758183381 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 620061630 0 0) 0x7feab00d0460 con 0x7feab00cf9d0 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.585+0000 7feaaeffd640 1 --2- 172.21.15.161:0/3758183381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab015e400 0x7feab00cf490 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feaa40097a0 tx=0x7feaa402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394823e279380507 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.585+0000 7feaadffb640 1 -- 172.21.15.161:0/3758183381 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4031830 con 0x7feab015e400 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.585+0000 7feaadffb640 1 -- 172.21.15.161:0/3758183381 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa4031990 con 0x7feab015e400 2024-03-23T14:35:56.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.585+0000 7feaadffb640 1 -- 172.21.15.161:0/3758183381 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa4031d20 con 0x7feab015e400 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.586+0000 7feab57f6640 1 -- 172.21.15.161:0/3758183381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab015e400 msgr2=0x7feab00cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.586+0000 7feab57f6640 1 --2- 172.21.15.161:0/3758183381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab015e400 0x7feab00cf490 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7feaa40097a0 tx=0x7feaa402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.586+0000 7feab57f6640 1 -- 172.21.15.161:0/3758183381 shutdown_connections 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.586+0000 7feab57f6640 1 --2- 172.21.15.161:0/3758183381 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00d2b40 0x7feab00b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.586+0000 7feab57f6640 1 --2- 172.21.15.161:0/3758183381 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab015e400 0x7feab00cf490 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.586+0000 7feab57f6640 1 -- 172.21.15.161:0/3758183381 >> 172.21.15.161:0/3758183381 conn(0x7feab009d190 msgr2=0x7feab00d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.586+0000 7feab57f6640 1 -- 172.21.15.161:0/3758183381 shutdown_connections 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.586+0000 7feab57f6640 1 -- 172.21.15.161:0/3758183381 wait complete. 2024-03-23T14:35:56.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.587+0000 7feab57f6640 1 Processor -- start 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.587+0000 7feab57f6640 1 -- start start 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feab57f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00cf9d0 0x7feab015e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feab57f6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab00d2b40 0x7feab00ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feab57f6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab015e400 0x7feab0150f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feab57f6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feab01514c0 con 0x7feab00d2b40 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feab57f6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feab00c93e0 con 0x7feab015e400 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feab57f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feab015edc0 con 0x7feab00cf9d0 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab00d2b40 0x7feab00ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab00d2b40 0x7feab00ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57360/0 (socket says 172.21.15.161:57360) 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaaeffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00cf9d0 0x7feab015e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 -- 172.21.15.161:0/861700253 learned_addr learned my addr 172.21.15.161:0/861700253 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaaf7fe640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab015e400 0x7feab0150f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 -- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00cf9d0 msgr2=0x7feab015e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00cf9d0 0x7feab015e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 -- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab015e400 msgr2=0x7feab0150f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab015e400 0x7feab0150f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 -- 172.21.15.161:0/861700253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa4009400 con 0x7feab00d2b40 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.588+0000 7feaae7fc640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab00d2b40 0x7feab00ca110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c500 tx=0x7feaa000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.589+0000 7fea977fe640 1 -- 172.21.15.161:0/861700253 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0007560 con 0x7feab00d2b40 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.589+0000 7fea977fe640 1 -- 172.21.15.161:0/861700253 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feaa00047e0 con 0x7feab00d2b40 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.589+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feab00c9630 con 0x7feab00d2b40 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.589+0000 7fea977fe640 1 -- 172.21.15.161:0/861700253 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feaa0004b90 con 0x7feab00d2b40 2024-03-23T14:35:56.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.589+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feab00c9bf0 con 0x7feab00d2b40 2024-03-23T14:35:56.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.590+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feab00d1450 con 0x7feab00d2b40 2024-03-23T14:35:56.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.590+0000 7fea977fe640 1 -- 172.21.15.161:0/861700253 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feaa0004cf0 con 0x7feab00d2b40 2024-03-23T14:35:56.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.590+0000 7fea977fe640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fea9c06b810 0x7fea9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.590+0000 7fea977fe640 1 -- 172.21.15.161:0/861700253 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7feaa00895e0 con 0x7feab00d2b40 2024-03-23T14:35:56.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.590+0000 7feaaeffd640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fea9c06b810 0x7fea9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.591+0000 7feaaeffd640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fea9c06b810 0x7fea9c06dcd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feaa4036000 tx=0x7feaa403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.594+0000 7fea977fe640 1 -- 172.21.15.161:0/861700253 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feab00d1450 con 0x7feab00d2b40 2024-03-23T14:35:56.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.801+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7feab00d1450 con 0x7feab00d2b40 2024-03-23T14:35:56.797 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-23T14:35:56.801+0000 7f8d40811640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T14:35:56.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.802+0000 7fea977fe640 1 -- 172.21.15.161:0/861700253 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v7) ==== 161+0+0 (secure 0 0 0) 0x7feab00d1450 con 0x7feab00d2b40 2024-03-23T14:35:56.798 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T14:35:56.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.805+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fea9c06b810 msgr2=0x7fea9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fea9c06b810 0x7fea9c06dcd0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7feaa4036000 tx=0x7feaa403c040 comp rx=0 tx=0).stop 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab00d2b40 msgr2=0x7feab00ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab00d2b40 0x7feab00ca110 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7feaa000c500 tx=0x7feaa000c9c0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 shutdown_connections 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fea9c06b810 0x7fea9c06dcd0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feab015e400 0x7feab0150f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feab00d2b40 0x7feab00ca110 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 --2- 172.21.15.161:0/861700253 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feab00cf9d0 0x7feab015e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 >> 172.21.15.161:0/861700253 conn(0x7feab009d190 msgr2=0x7feab00b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 shutdown_connections 2024-03-23T14:35:56.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.806+0000 7feab57f6640 1 -- 172.21.15.161:0/861700253 wait complete. 2024-03-23T14:35:56.804 INFO:tasks.ceph.mds.b.smithi183.stderr: -4107> 2024-03-23T14:35:06.723+0000 7f9a0a91b640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-23T14:35:56.804 INFO:tasks.ceph.mds.b.smithi183.stderr: -2905> 2024-03-23T14:35:17.735+0000 7f9a0490f640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T14:35:56.813 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T14:35:56.830 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-23T14:35:56.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.923+0000 7ff675f56640 1 Processor -- start 2024-03-23T14:35:56.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.923+0000 7ff675f56640 1 -- start start 2024-03-23T14:35:56.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff675f56640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff670145520 0x7ff670160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff675f56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6700cfee0 0x7ff6700d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff675f56640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff670070c20 0x7ff6700710b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff675f56640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff6700d0880 con 0x7ff670145520 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff675f56640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7ff6701608c0 con 0x7ff670070c20 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff675f56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff6700ce410 con 0x7ff6700cfee0 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff670145520 0x7ff670160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff670145520 0x7ff670160380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57368/0 (socket says 172.21.15.161:57368) 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66f7fe640 1 -- 172.21.15.161:0/3515337717 learned_addr learned my addr 172.21.15.161:0/3515337717 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66ffff640 1 --2- 172.21.15.161:0/3515337717 >> v2:172.21.15.183:3300/0 conn(0x7ff670070c20 0x7ff6700710b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66effd640 1 --2- 172.21.15.161:0/3515337717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6700cfee0 0x7ff6700d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66f7fe640 1 -- 172.21.15.161:0/3515337717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6700cfee0 msgr2=0x7ff6700d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66f7fe640 1 --2- 172.21.15.161:0/3515337717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6700cfee0 0x7ff6700d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66f7fe640 1 -- 172.21.15.161:0/3515337717 >> v2:172.21.15.183:3300/0 conn(0x7ff670070c20 msgr2=0x7ff6700710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:56.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66f7fe640 1 --2- 172.21.15.161:0/3515337717 >> v2:172.21.15.183:3300/0 conn(0x7ff670070c20 0x7ff6700710b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.924+0000 7ff66f7fe640 1 -- 172.21.15.161:0/3515337717 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6700ce6f0 con 0x7ff670145520 2024-03-23T14:35:56.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff66f7fe640 1 --2- 172.21.15.161:0/3515337717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff670160380 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff67015c380 tx=0x7ff66002b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d85ddf02945aff6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff66e7fc640 1 -- 172.21.15.161:0/3515337717 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6600317e0 con 0x7ff670145520 2024-03-23T14:35:56.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff66e7fc640 1 -- 172.21.15.161:0/3515337717 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff660031940 con 0x7ff670145520 2024-03-23T14:35:56.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff66e7fc640 1 -- 172.21.15.161:0/3515337717 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff660031cd0 con 0x7ff670145520 2024-03-23T14:35:56.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff675f56640 1 -- 172.21.15.161:0/3515337717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 msgr2=0x7ff670160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff675f56640 1 --2- 172.21.15.161:0/3515337717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff670160380 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ff67015c380 tx=0x7ff66002b880 comp rx=0 tx=0).stop 2024-03-23T14:35:56.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff675f56640 1 -- 172.21.15.161:0/3515337717 shutdown_connections 2024-03-23T14:35:56.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff675f56640 1 --2- 172.21.15.161:0/3515337717 >> v2:172.21.15.183:3300/0 conn(0x7ff670070c20 0x7ff6700710b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff675f56640 1 --2- 172.21.15.161:0/3515337717 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff6700cfee0 0x7ff6700d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff675f56640 1 --2- 172.21.15.161:0/3515337717 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff670160380 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.925+0000 7ff675f56640 1 -- 172.21.15.161:0/3515337717 >> 172.21.15.161:0/3515337717 conn(0x7ff67006b4a0 msgr2=0x7ff67006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.926+0000 7ff675f56640 1 -- 172.21.15.161:0/3515337717 shutdown_connections 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.926+0000 7ff675f56640 1 -- 172.21.15.161:0/3515337717 wait complete. 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.927+0000 7ff675f56640 1 Processor -- start 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.927+0000 7ff675f56640 1 -- start start 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff675f56640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff670070c20 0x7ff67013b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff675f56640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6700cfee0 0x7ff6701410c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff675f56640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff67015eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff675f56640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff67015d1e0 con 0x7ff670145520 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff675f56640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff67015d4c0 con 0x7ff6700cfee0 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff675f56640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff67015d7a0 con 0x7ff670070c20 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff67015eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff67015eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57374/0 (socket says 172.21.15.161:57374) 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 -- 172.21.15.161:0/2171796537 learned_addr learned my addr 172.21.15.161:0/2171796537 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66f7fe640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff670070c20 0x7ff67013b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66effd640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6700cfee0 0x7ff6701410c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 -- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff670070c20 msgr2=0x7ff67013b5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff670070c20 0x7ff67013b5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 -- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6700cfee0 msgr2=0x7ff6701410c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6700cfee0 0x7ff6701410c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 -- 172.21.15.161:0/2171796537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6600094d0 con 0x7ff670145520 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff66ffff640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff67015eef0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff668007e60 tx=0x7ff66800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff657fff640 1 -- 172.21.15.161:0/2171796537 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff668014420 con 0x7ff670145520 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff657fff640 1 -- 172.21.15.161:0/2171796537 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6680145a0 con 0x7ff670145520 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff657fff640 1 -- 172.21.15.161:0/2171796537 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff668014950 con 0x7ff670145520 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff67015da80 con 0x7ff670145520 2024-03-23T14:35:56.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.928+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff67015cd90 con 0x7ff670145520 2024-03-23T14:35:56.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.929+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff67015cfb0 con 0x7ff670145520 2024-03-23T14:35:56.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.930+0000 7ff657fff640 1 -- 172.21.15.161:0/2171796537 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff668014ab0 con 0x7ff670145520 2024-03-23T14:35:56.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.930+0000 7ff657fff640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff65c06b810 0x7ff65c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:56.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.930+0000 7ff657fff640 1 -- 172.21.15.161:0/2171796537 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5911+0+0 (secure 0 0 0) 0x7ff66808a9e0 con 0x7ff670145520 2024-03-23T14:35:56.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.930+0000 7ff66f7fe640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff65c06b810 0x7ff65c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:56.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.931+0000 7ff66f7fe640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff65c06b810 0x7ff65c06dcd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff660036000 tx=0x7ff66003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:56.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:56.934+0000 7ff657fff640 1 -- 172.21.15.161:0/2171796537 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff668055470 con 0x7ff670145520 2024-03-23T14:35:57.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.141+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff67013c0a0 con 0x7ff670145520 2024-03-23T14:35:57.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.804+0000 7ff657fff640 1 -- 172.21.15.161:0/2171796537 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v8) ==== 104+0+0 (secure 0 0 0) 0x7ff67013c0a0 con 0x7ff670145520 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff65c06b810 msgr2=0x7ff65c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff65c06b810 0x7ff65c06dcd0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7ff660036000 tx=0x7ff66003d040 comp rx=0 tx=0).stop 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 msgr2=0x7ff67015eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff67015eef0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7ff668007e60 tx=0x7ff66800e4b0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 shutdown_connections 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff65c06b810 0x7ff65c06dcd0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff670145520 0x7ff67015eef0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff6700cfee0 0x7ff6701410c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 --2- 172.21.15.161:0/2171796537 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff670070c20 0x7ff67013b5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 >> 172.21.15.161:0/2171796537 conn(0x7ff67006b4a0 msgr2=0x7ff67015a3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:57.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 shutdown_connections 2024-03-23T14:35:57.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.807+0000 7ff675f56640 1 -- 172.21.15.161:0/2171796537 wait complete. 2024-03-23T14:35:57.816 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T14:35:57.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.929+0000 7f4bb3e9d640 1 Processor -- start 2024-03-23T14:35:57.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.929+0000 7f4bb3e9d640 1 -- start start 2024-03-23T14:35:57.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb3e9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4bac144660 0x7f4bac06e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:57.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb3e9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0cfce0 0x7f4bac0d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:57.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb3e9d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f4bac0b2b10 0x7f4bac0b2fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:57.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb3e9d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f4bac06d0e0 con 0x7f4bac144660 2024-03-23T14:35:57.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb3e9d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f4bac0d0660 con 0x7f4bac0b2b10 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb3e9d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4bac144a40 con 0x7f4bac0cfce0 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1c39640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4bac144660 0x7f4bac06e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1c39640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f4bac144660 0x7f4bac06e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57376/0 (socket says 172.21.15.161:57376) 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1c39640 1 -- 172.21.15.161:0/2836835363 learned_addr learned my addr 172.21.15.161:0/2836835363 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb243a640 1 --2- 172.21.15.161:0/2836835363 >> v2:172.21.15.183:3300/0 conn(0x7f4bac0b2b10 0x7f4bac0b2fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1438640 1 --2- 172.21.15.161:0/2836835363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0cfce0 0x7f4bac0d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1c39640 1 -- 172.21.15.161:0/2836835363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0cfce0 msgr2=0x7f4bac0d0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1c39640 1 --2- 172.21.15.161:0/2836835363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0cfce0 0x7f4bac0d0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1c39640 1 -- 172.21.15.161:0/2836835363 >> v2:172.21.15.183:3300/0 conn(0x7f4bac0b2b10 msgr2=0x7f4bac0b2fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1c39640 1 --2- 172.21.15.161:0/2836835363 >> v2:172.21.15.183:3300/0 conn(0x7f4bac0b2b10 0x7f4bac0b2fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb1c39640 1 -- 172.21.15.161:0/2836835363 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac06d330 con 0x7f4bac144660 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.930+0000 7f4bb243a640 1 --2- 172.21.15.161:0/2836835363 >> v2:172.21.15.183:3300/0 conn(0x7f4bac0b2b10 0x7f4bac0b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:35:57.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.931+0000 7f4bb1c39640 1 --2- 172.21.15.161:0/2836835363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac06e0f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0ceb00 tx=0x7f4ba402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6ca8c61dc6530f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.931+0000 7f4bb0c37640 1 -- 172.21.15.161:0/2836835363 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba402ca70 con 0x7f4bac144660 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.931+0000 7f4bb0c37640 1 -- 172.21.15.161:0/2836835363 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba402cbd0 con 0x7f4bac144660 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.931+0000 7f4bb0c37640 1 -- 172.21.15.161:0/2836835363 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba40343f0 con 0x7f4bac144660 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/2836835363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 msgr2=0x7f4bac06e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/2836835363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac06e0f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4bac0ceb00 tx=0x7f4ba402bed0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/2836835363 shutdown_connections 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/2836835363 >> v2:172.21.15.183:3300/0 conn(0x7f4bac0b2b10 0x7f4bac0b2fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/2836835363 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0cfce0 0x7f4bac0d0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/2836835363 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac06e0f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/2836835363 >> 172.21.15.161:0/2836835363 conn(0x7f4bac06b260 msgr2=0x7f4bac06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:57.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/2836835363 shutdown_connections 2024-03-23T14:35:57.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.932+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/2836835363 wait complete. 2024-03-23T14:35:57.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.933+0000 7f4bb3e9d640 1 Processor -- start 2024-03-23T14:35:57.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.933+0000 7f4bb3e9d640 1 -- start start 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb3e9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0b2b10 0x7f4bac13e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb3e9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bac0cfce0 0x7f4bac141420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb3e9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac158330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb3e9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac142ab0 con 0x7f4bac144660 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb3e9d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4bac141ea0 con 0x7f4bac0cfce0 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb3e9d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4bac142180 con 0x7f4bac0b2b10 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac158330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac158330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57380/0 (socket says 172.21.15.161:57380) 2024-03-23T14:35:57.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 -- 172.21.15.161:0/969498152 learned_addr learned my addr 172.21.15.161:0/969498152 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:57.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb1438640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bac0cfce0 0x7f4bac141420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:57.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb1c39640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0b2b10 0x7f4bac13e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:57.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 -- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0b2b10 msgr2=0x7f4bac13e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:57.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0b2b10 0x7f4bac13e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 -- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bac0cfce0 msgr2=0x7f4bac141420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:57.936 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bac0cfce0 0x7f4bac141420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:57.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 -- 172.21.15.161:0/969498152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ba4009b30 con 0x7f4bac144660 2024-03-23T14:35:57.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb1c39640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0b2b10 0x7f4bac13e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:35:57.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb1438640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bac0cfce0 0x7f4bac141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:35:57.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb243a640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac158330 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800e5e0 tx=0x7f4ba800eaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:57.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4ba27fc640 1 -- 172.21.15.161:0/969498152 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8004230 con 0x7f4bac144660 2024-03-23T14:35:57.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4ba27fc640 1 -- 172.21.15.161:0/969498152 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ba8004390 con 0x7f4bac144660 2024-03-23T14:35:57.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac142460 con 0x7f4bac144660 2024-03-23T14:35:57.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4ba27fc640 1 -- 172.21.15.161:0/969498152 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ba8003850 con 0x7f4bac144660 2024-03-23T14:35:57.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.934+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bac1f1d40 con 0x7f4bac144660 2024-03-23T14:35:57.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.935+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bac142c60 con 0x7f4bac144660 2024-03-23T14:35:57.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.936+0000 7f4ba27fc640 1 -- 172.21.15.161:0/969498152 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4ba80039b0 con 0x7f4bac144660 2024-03-23T14:35:57.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.936+0000 7f4ba27fc640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4b9006b840 0x7f4b9006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:57.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.936+0000 7f4ba27fc640 1 -- 172.21.15.161:0/969498152 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5911+0+0 (secure 0 0 0) 0x7f4ba8089e00 con 0x7f4bac144660 2024-03-23T14:35:57.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.937+0000 7f4bb1c39640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4b9006b840 0x7f4b9006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:57.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.937+0000 7f4bb1c39640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4b9006b840 0x7f4b9006dd00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4035000 tx=0x7f4ba4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:57.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:57.941+0000 7f4ba27fc640 1 -- 172.21.15.161:0/969498152 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4bac142c60 con 0x7f4bac144660 2024-03-23T14:35:58.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.111+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f4bac04f4e0 con 0x7f4bac144660 2024-03-23T14:35:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.877+0000 7f4ba27fc640 1 -- 172.21.15.161:0/969498152 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v24) ==== 181+0+0 (secure 0 0 0) 0x7f4ba80200d0 con 0x7f4bac144660 2024-03-23T14:35:58.873 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-23T14:35:58.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.882+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4b9006b840 msgr2=0x7f4b9006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:58.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.882+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4b9006b840 0x7f4b9006dd00 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4035000 tx=0x7f4ba4030000 comp rx=0 tx=0).stop 2024-03-23T14:35:58.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.882+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 msgr2=0x7f4bac158330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:58.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.882+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac158330 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f4ba800e5e0 tx=0x7f4ba800eaa0 comp rx=0 tx=0).stop 2024-03-23T14:35:58.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.883+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 shutdown_connections 2024-03-23T14:35:58.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.883+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f4b9006b840 0x7f4b9006dd00 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:58.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.883+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f4bac144660 0x7f4bac158330 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:58.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.883+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4bac0cfce0 0x7f4bac141420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:58.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.883+0000 7f4bb3e9d640 1 --2- 172.21.15.161:0/969498152 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4bac0b2b10 0x7f4bac13e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:58.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.883+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 >> 172.21.15.161:0/969498152 conn(0x7f4bac06b260 msgr2=0x7f4bac0cd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:58.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.883+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 shutdown_connections 2024-03-23T14:35:58.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:58.883+0000 7f4bb3e9d640 1 -- 172.21.15.161:0/969498152 wait complete. 2024-03-23T14:35:58.893 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T14:35:59.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.006+0000 7fa0b5181640 1 Processor -- start 2024-03-23T14:35:59.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.006+0000 7fa0b5181640 1 -- start start 2024-03-23T14:35:59.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0b5181640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa0b01446e0 0x7fa0b006de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:59.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0b5181640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b0144ac0 con 0x7fa0b006e380 2024-03-23T14:35:59.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0b5181640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0b015b750 con 0x7fa0b0071f70 2024-03-23T14:35:59.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0b5181640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa0b006e820 con 0x7fa0b01446e0 2024-03-23T14:35:59.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0aed9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa0b01446e0 0x7fa0b006de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:59.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0ae59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa0b006e380 0x7fa0b0071860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37404/0 (socket says 172.21.15.161:37404) 2024-03-23T14:35:59.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0aed9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa0b01446e0 0x7fa0b006de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57388/0 (socket says 172.21.15.161:57388) 2024-03-23T14:35:59.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0ae59c640 1 -- 172.21.15.161:0/4274007455 learned_addr learned my addr 172.21.15.161:0/4274007455 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:35:59.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0aed9d640 1 -- 172.21.15.161:0/4274007455 >> v1:172.21.15.183:6789/0 conn(0x7fa0b0071f70 legacy=0x7fa0b015bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:59.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0aed9d640 1 -- 172.21.15.161:0/4274007455 >> v1:172.21.15.161:6789/0 conn(0x7fa0b006e380 legacy=0x7fa0b0071860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:59.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0aed9d640 1 -- 172.21.15.161:0/4274007455 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0b006eb00 con 0x7fa0b01446e0 2024-03-23T14:35:59.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.007+0000 7fa0add9b640 1 -- 172.21.15.161:0/4274007455 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 899217341 0 0) 0x7fa0b0144ac0 con 0x7fa0b006e380 2024-03-23T14:35:59.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.008+0000 7fa0aed9d640 1 --2- 172.21.15.161:0/4274007455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b006de40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009d90 tx=0x7fa0a002bf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a187213a5712fba4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:59.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.008+0000 7fa0add9b640 1 -- 172.21.15.161:0/4274007455 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a002cd00 con 0x7fa0b01446e0 2024-03-23T14:35:59.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.008+0000 7fa0add9b640 1 -- 172.21.15.161:0/4274007455 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a002ce60 con 0x7fa0b01446e0 2024-03-23T14:35:59.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.008+0000 7fa0add9b640 1 -- 172.21.15.161:0/4274007455 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a0034660 con 0x7fa0b01446e0 2024-03-23T14:35:59.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.009+0000 7fa0b5181640 1 -- 172.21.15.161:0/4274007455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 msgr2=0x7fa0b006de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:59.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.009+0000 7fa0b5181640 1 --2- 172.21.15.161:0/4274007455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b006de40 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009d90 tx=0x7fa0a002bf50 comp rx=0 tx=0).stop 2024-03-23T14:35:59.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.009+0000 7fa0b5181640 1 -- 172.21.15.161:0/4274007455 shutdown_connections 2024-03-23T14:35:59.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.009+0000 7fa0b5181640 1 --2- 172.21.15.161:0/4274007455 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b006de40 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:59.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.009+0000 7fa0b5181640 1 -- 172.21.15.161:0/4274007455 >> 172.21.15.161:0/4274007455 conn(0x7fa0b006b080 msgr2=0x7fa0b006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:59.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.009+0000 7fa0b5181640 1 -- 172.21.15.161:0/4274007455 shutdown_connections 2024-03-23T14:35:59.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.009+0000 7fa0b5181640 1 -- 172.21.15.161:0/4274007455 wait complete. 2024-03-23T14:35:59.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.010+0000 7fa0b5181640 1 Processor -- start 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.010+0000 7fa0b5181640 1 -- start start 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0b5181640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0b006e380 0x7fa0b0138950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0b5181640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0b0071f70 0x7fa0b0138e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0b5181640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b01efdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0b5181640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b0154120 con 0x7fa0b01446e0 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0b5181640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa0b01f0dc0 con 0x7fa0b006e380 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0b5181640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa0b01f10a0 con 0x7fa0b0071f70 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0af59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b01efdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0af59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b01efdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57402/0 (socket says 172.21.15.161:57402) 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0af59e640 1 -- 172.21.15.161:0/164986214 learned_addr learned my addr 172.21.15.161:0/164986214 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0aed9d640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0b006e380 0x7fa0b0138950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0ae59c640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0b0071f70 0x7fa0b0138e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:59.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0af59e640 1 -- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0b0071f70 msgr2=0x7fa0b0138e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0af59e640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0b0071f70 0x7fa0b0138e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0af59e640 1 -- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0b006e380 msgr2=0x7fa0b0138950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0af59e640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0b006e380 0x7fa0b0138950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.011+0000 7fa0af59e640 1 -- 172.21.15.161:0/164986214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a0009a70 con 0x7fa0b01446e0 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.012+0000 7fa0af59e640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b01efdd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8001eb0 tx=0x7fa0a800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.012+0000 7fa0977fe640 1 -- 172.21.15.161:0/164986214 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a80052d0 con 0x7fa0b01446e0 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.012+0000 7fa0977fe640 1 -- 172.21.15.161:0/164986214 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a8005430 con 0x7fa0b01446e0 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.012+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0b01f1380 con 0x7fa0b01446e0 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.012+0000 7fa0977fe640 1 -- 172.21.15.161:0/164986214 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8003810 con 0x7fa0b01446e0 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.012+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0b01f1940 con 0x7fa0b01446e0 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.013+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa0b00735e0 con 0x7fa0b01446e0 2024-03-23T14:35:59.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.013+0000 7fa0977fe640 1 -- 172.21.15.161:0/164986214 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa0a8003970 con 0x7fa0b01446e0 2024-03-23T14:35:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.014+0000 7fa0977fe640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa09c06b810 0x7fa09c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:35:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.014+0000 7fa0977fe640 1 -- 172.21.15.161:0/164986214 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 5473+0+0 (secure 0 0 0) 0x7fa0a80896f0 con 0x7fa0b01446e0 2024-03-23T14:35:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.014+0000 7fa0aed9d640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa09c06b810 0x7fa09c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:35:59.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.014+0000 7fa0aed9d640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa09c06b810 0x7fa09c06dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0036000 tx=0x7fa0a0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:35:59.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.018+0000 7fa0977fe640 1 -- 172.21.15.161:0/164986214 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa0a80541f0 con 0x7fa0b01446e0 2024-03-23T14:35:59.185 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.189+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa0b0168a20 con 0x7fa0b01446e0 2024-03-23T14:35:59.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.879+0000 7fa0977fe640 1 -- 172.21.15.161:0/164986214 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v25) ==== 169+0+0 (secure 0 0 0) 0x7fa0a80578b0 con 0x7fa0b01446e0 2024-03-23T14:35:59.875 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-23T14:35:59.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa09c06b810 msgr2=0x7fa09c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa09c06b810 0x7fa09c06dcd0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0036000 tx=0x7fa0a0030000 comp rx=0 tx=0).stop 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 msgr2=0x7fa0b01efdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b01efdd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa0a8001eb0 tx=0x7fa0a800c890 comp rx=0 tx=0).stop 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 shutdown_connections 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa09c06b810 0x7fa09c06dcd0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0b01446e0 0x7fa0b01efdd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0b0071f70 0x7fa0b0138e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 --2- 172.21.15.161:0/164986214 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0b006e380 0x7fa0b0138950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 >> 172.21.15.161:0/164986214 conn(0x7fa0b006b080 msgr2=0x7fa0b006b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 shutdown_connections 2024-03-23T14:35:59.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:35:59.881+0000 7fa0b5181640 1 -- 172.21.15.161:0/164986214 wait complete. 2024-03-23T14:35:59.894 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T14:35:59.894 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T14:35:59.895 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T14:35:59.895 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:35:59.895 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T14:35:59.895 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T14:35:59.895 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T14:35:59.895 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:35:59.896 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T14:35:59.896 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T14:35:59.896 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T14:35:59.896 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:35:59.896 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T14:35:59.896 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T14:35:59.896 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T14:35:59.896 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:35:59.905 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T14:35:59.906 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T14:35:59.907 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T14:35:59.907 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T14:35:59.908 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T14:35:59.908 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T14:35:59.909 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T14:35:59.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T14:35:59.948 INFO:tasks.ceph.mds.a:Started 2024-03-23T14:35:59.948 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:59.949 INFO:tasks.ceph.mds.c:Started 2024-03-23T14:35:59.949 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:59.951 INFO:tasks.ceph.mds.d:Started 2024-03-23T14:35:59.952 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:59.952 INFO:tasks.ceph.mds.b:Started 2024-03-23T14:35:59.952 DEBUG:teuthology.parallel:result is None 2024-03-23T14:35:59.953 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T14:36:00.026 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-23T14:36:00.026 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-23T14:36:00.035 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-23T14:36:00.035 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-23T14:36:00.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.064+0000 7fd42e3e2640 1 Processor -- start 2024-03-23T14:36:00.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.064+0000 7fd42e3e2640 1 -- start start 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd42e3e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280d2b40 0x7fd4280b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd42e3e2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4280d0460 con 0x7fd42815e400 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd42e3e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4280d06b0 con 0x7fd4280d1470 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd42e3e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd4280d1050 con 0x7fd4280d2b40 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd427fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd42815e400 0x7fd4280cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:37414/0 (socket says 172.21.15.161:37414) 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd427fff640 1 -- 172.21.15.161:0/1170951821 learned_addr learned my addr 172.21.15.161:0/1170951821 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3143071077 0 0) 0x7fd4280d0460 con 0x7fd42815e400 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd410003520 con 0x7fd42815e400 2024-03-23T14:36:00.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd42c97f640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280d2b40 0x7fd4280b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 873489299 0 0) 0x7fd410003520 con 0x7fd42815e400 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280d2b40 msgr2=0x7fd4280b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd426ffd640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280d2b40 0x7fd4280b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 >> v1:172.21.15.183:6789/0 conn(0x7fd4280d1470 legacy=0x7fd4280d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.065+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4280d0990 con 0x7fd42815e400 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fd418001670 con 0x7fd42815e400 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 >> v1:172.21.15.161:6789/0 conn(0x7fd42815e400 legacy=0x7fd4280cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd426ffd640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280d2b40 0x7fd4100039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd426ffd640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410003f70 0x7fd4100047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd426ffd640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd410004d00 0x7fd410005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd4280d1050 con 0x7fd410003f70 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd41000bac0 con 0x7fd410004d00 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd41000bc80 con 0x7fd4280d2b40 2024-03-23T14:36:00.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd4277fe640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410003f70 0x7fd4100047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd42c97f640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd410004d00 0x7fd410005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd427fff640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280d2b40 0x7fd4100039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd42c97f640 1 -- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280d2b40 msgr2=0x7fd4100039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd42c97f640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280d2b40 0x7fd4100039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd42c97f640 1 -- 172.21.15.161:0/1170951821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410003f70 msgr2=0x7fd4100047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd42c97f640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410003f70 0x7fd4100047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.066+0000 7fd42c97f640 1 -- 172.21.15.161:0/1170951821 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd410003520 con 0x7fd410004d00 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.067+0000 7fd42c97f640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd410004d00 0x7fd410005180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd420009fc0 tx=0x7fd4100057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f565d16c899cdcb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.067+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42002cc20 con 0x7fd410004d00 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.067+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd42000f040 con 0x7fd410004d00 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.067+0000 7fd426ffd640 1 -- 172.21.15.161:0/1170951821 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd420034440 con 0x7fd410004d00 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.068+0000 7fd42e3e2640 1 -- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd410004d00 msgr2=0x7fd410005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.068+0000 7fd42e3e2640 1 --2- 172.21.15.161:0/1170951821 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd410004d00 0x7fd410005180 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd420009fc0 tx=0x7fd4100057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.068+0000 7fd42c97f640 1 -- 172.21.15.161:0/1170951821 reap_dead start 2024-03-23T14:36:00.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.068+0000 7fd42e3e2640 1 -- 172.21.15.161:0/1170951821 shutdown_connections 2024-03-23T14:36:00.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.068+0000 7fd42e3e2640 1 -- 172.21.15.161:0/1170951821 >> 172.21.15.161:0/1170951821 conn(0x7fd42809d190 msgr2=0x7fd4280d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:00.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.068+0000 7fd42e3e2640 1 -- 172.21.15.161:0/1170951821 shutdown_connections 2024-03-23T14:36:00.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.068+0000 7fd42e3e2640 1 -- 172.21.15.161:0/1170951821 wait complete. 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.069+0000 7fd42e3e2640 1 Processor -- start 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.069+0000 7fd42e3e2640 1 -- start start 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd42e3e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410004d00 0x7fd428154db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd42e3e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4281552f0 0x7fd4280cbf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd42e3e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280ca960 0x7fd4280cc470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd427fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410004d00 0x7fd428154db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd42e3e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd4280caf20 con 0x7fd410004d00 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd42e3e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd428154bc0 con 0x7fd4281552f0 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd427fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410004d00 0x7fd428154db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:57450/0 (socket says 172.21.15.161:57450) 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd427fff640 1 -- 172.21.15.161:0/3061265125 learned_addr learned my addr 172.21.15.161:0/3061265125 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:00.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd4280cb170 con 0x7fd4280ca960 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd4277fe640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4281552f0 0x7fd4280cbf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.070+0000 7fd42c97f640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280ca960 0x7fd4280cc470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd427fff640 1 -- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280ca960 msgr2=0x7fd4280cc470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd427fff640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280ca960 0x7fd4280cc470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd427fff640 1 -- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4281552f0 msgr2=0x7fd4280cbf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd427fff640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4281552f0 0x7fd4280cbf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd427fff640 1 -- 172.21.15.161:0/3061265125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4280cb450 con 0x7fd410004d00 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd427fff640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410004d00 0x7fd428154db0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd4180072f0 tx=0x7fd4180076f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd424ff9640 1 -- 172.21.15.161:0/3061265125 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418007760 con 0x7fd410004d00 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd424ff9640 1 -- 172.21.15.161:0/3061265125 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41800f9c0 con 0x7fd410004d00 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd424ff9640 1 -- 172.21.15.161:0/3061265125 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41800fd50 con 0x7fd410004d00 2024-03-23T14:36:00.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4280cb730 con 0x7fd410004d00 2024-03-23T14:36:00.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.071+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4280b53e0 con 0x7fd410004d00 2024-03-23T14:36:00.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.072+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4280cbc50 con 0x7fd410004d00 2024-03-23T14:36:00.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.072+0000 7fd424ff9640 1 -- 172.21.15.161:0/3061265125 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd41801d460 con 0x7fd410004d00 2024-03-23T14:36:00.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.073+0000 7fd424ff9640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41406b810 0x7fd41406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:00.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.073+0000 7fd424ff9640 1 -- 172.21.15.161:0/3061265125 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 4943+0+0 (secure 0 0 0) 0x7fd418089f50 con 0x7fd410004d00 2024-03-23T14:36:00.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.073+0000 7fd4277fe640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41406b810 0x7fd41406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:00.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.073+0000 7fd4277fe640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41406b810 0x7fd41406dcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd4281556d0 tx=0x7fd41c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:00.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.077+0000 7fd424ff9640 1 -- 172.21.15.161:0/3061265125 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4280cbc50 con 0x7fd410004d00 2024-03-23T14:36:00.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.259+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7fd4280cbc50 con 0x7fd410004d00 2024-03-23T14:36:00.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.900+0000 7fd424ff9640 1 -- 172.21.15.161:0/3061265125 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v27) ==== 97+0+0 (secure 0 0 0) 0x7fd4280cbc50 con 0x7fd410004d00 2024-03-23T14:36:00.897 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-23T14:36:00.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41406b810 msgr2=0x7fd41406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41406b810 0x7fd41406dcd0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd4281556d0 tx=0x7fd41c009040 comp rx=0 tx=0).stop 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410004d00 msgr2=0x7fd428154db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410004d00 0x7fd428154db0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd4180072f0 tx=0x7fd4180076f0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 shutdown_connections 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41406b810 0x7fd41406dcd0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd4280ca960 0x7fd4280cc470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4281552f0 0x7fd4280cbf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 --2- 172.21.15.161:0/3061265125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd410004d00 0x7fd428154db0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 >> 172.21.15.161:0/3061265125 conn(0x7fd42809d190 msgr2=0x7fd4280b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 shutdown_connections 2024-03-23T14:36:00.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:00.903+0000 7fd42e3e2640 1 -- 172.21.15.161:0/3061265125 wait complete. 2024-03-23T14:36:00.913 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T14:36:01.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.017+0000 7fb1d6d6c640 1 Processor -- start 2024-03-23T14:36:01.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.017+0000 7fb1d6d6c640 1 -- start start 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.018+0000 7fb1d6d6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 0x7fb1d006e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.018+0000 7fb1d6d6c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d015be70 con 0x7fb1d0072100 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.018+0000 7fb1d6d6c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1d015c150 con 0x7fb1d006ead0 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.018+0000 7fb1d6d6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb1d006ef80 con 0x7fb1d00724e0 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.018+0000 7fb1d4b08640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb1d0072100 0x7fb1d015b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38900/0 (socket says 172.21.15.161:38900) 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.018+0000 7fb1d4b08640 1 -- 172.21.15.161:0/2633527853 learned_addr learned my addr 172.21.15.161:0/2633527853 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2404433056 0 0) 0x7fb1d015be70 con 0x7fb1d0072100 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cffff640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 0x7fb1d006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1c0003520 con 0x7fb1d0072100 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1994155455 0 0) 0x7fb1c0003520 con 0x7fb1d0072100 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 msgr2=0x7fb1d006e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 0x7fb1d006e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 >> v1:172.21.15.183:6789/0 conn(0x7fb1d006ead0 legacy=0x7fb1d00b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d00b2c70 con 0x7fb1d0072100 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fb1c4001670 con 0x7fb1d0072100 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 >> v1:172.21.15.161:6789/0 conn(0x7fb1d0072100 legacy=0x7fb1d015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1c00039a0 0x7fb1c0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1c00047c0 0x7fb1c0004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 0x7fb1c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c000ba30 con 0x7fb1c00039a0 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c000dd80 con 0x7fb1c00047c0 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c000df40 con 0x7fb1d00724e0 2024-03-23T14:36:01.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1d4b08640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1c00039a0 0x7fb1c0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1cffff640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1c00047c0 0x7fb1c0004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1d5309640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 0x7fb1c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1d4b08640 1 -- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 msgr2=0x7fb1c0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1d4b08640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 0x7fb1c0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1d4b08640 1 -- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1c00047c0 msgr2=0x7fb1c0004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1d4b08640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1c00047c0 0x7fb1c0004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1d4b08640 1 -- 172.21.15.161:0/2633527853 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c0003520 con 0x7fb1c00039a0 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.019+0000 7fb1d5309640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1d00724e0 0x7fb1c0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.020+0000 7fb1d4b08640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1c00039a0 0x7fb1c0003dc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c210 tx=0x7fb1c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5fb555b97583cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.020+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c4005ad0 con 0x7fb1c00039a0 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.020+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1c4005c50 con 0x7fb1c00039a0 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.020+0000 7fb1cf7fe640 1 -- 172.21.15.161:0/2633527853 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1c402c870 con 0x7fb1c00039a0 2024-03-23T14:36:01.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.021+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/2633527853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1c00039a0 msgr2=0x7fb1c0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.021+0000 7fb1d6d6c640 1 --2- 172.21.15.161:0/2633527853 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1c00039a0 0x7fb1c0003dc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fb1c400c210 tx=0x7fb1c00057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.021+0000 7fb1d5309640 1 -- 172.21.15.161:0/2633527853 reap_dead start 2024-03-23T14:36:01.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.021+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/2633527853 shutdown_connections 2024-03-23T14:36:01.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.021+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/2633527853 >> 172.21.15.161:0/2633527853 conn(0x7fb1d00b17f0 msgr2=0x7fb1d00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:01.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.021+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/2633527853 shutdown_connections 2024-03-23T14:36:01.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.021+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/2633527853 wait complete. 2024-03-23T14:36:01.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.022+0000 7fb1d6d6c640 1 Processor -- start 2024-03-23T14:36:01.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.022+0000 7fb1d6d6c640 1 -- start start 2024-03-23T14:36:01.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1d6d6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1c00039a0 0x7fb1d014da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1d6d6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1d00ac620 0x7fb1d0159fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1d6d6c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1d015a4e0 0x7fb1d01581d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1d6d6c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d014df60 con 0x7fb1d00ac620 2024-03-23T14:36:01.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1d6d6c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb1d015a960 con 0x7fb1d015a4e0 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1d6d6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb1d00aa150 con 0x7fb1c00039a0 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1d00ac620 0x7fb1d0159fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1d00ac620 0x7fb1d0159fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38074/0 (socket says 172.21.15.161:38074) 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 -- 172.21.15.161:0/3454964332 learned_addr learned my addr 172.21.15.161:0/3454964332 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1d4b08640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1c00039a0 0x7fb1d014da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1d5309640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1d015a4e0 0x7fb1d01581d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 -- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1c00039a0 msgr2=0x7fb1d014da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1c00039a0 0x7fb1d014da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 -- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1d015a4e0 msgr2=0x7fb1d01581d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1d015a4e0 0x7fb1d01581d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 -- 172.21.15.161:0/3454964332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1d00a9540 con 0x7fb1d00ac620 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.023+0000 7fb1cffff640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1d00ac620 0x7fb1d0159fa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0119a0 tx=0x7fb1bc011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.024+0000 7fb1cd7fa640 1 -- 172.21.15.161:0/3454964332 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc00faf0 con 0x7fb1d00ac620 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.024+0000 7fb1cd7fa640 1 -- 172.21.15.161:0/3454964332 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1bc00fc50 con 0x7fb1d00ac620 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.024+0000 7fb1cd7fa640 1 -- 172.21.15.161:0/3454964332 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc014440 con 0x7fb1d00ac620 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.024+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1d00a9820 con 0x7fb1d00ac620 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.024+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1d00a9de0 con 0x7fb1d00ac620 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.025+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1d00735e0 con 0x7fb1d00ac620 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.025+0000 7fb1cd7fa640 1 -- 172.21.15.161:0/3454964332 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb1bc0145a0 con 0x7fb1d00ac620 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.025+0000 7fb1cd7fa640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb1b406b810 0x7fb1b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.026+0000 7fb1cd7fa640 1 -- 172.21.15.161:0/3454964332 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fb1bc08a5d0 con 0x7fb1d00ac620 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.026+0000 7fb1d4b08640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb1b406b810 0x7fb1b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.026+0000 7fb1d4b08640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb1b406b810 0x7fb1b406dcd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4034000 tx=0x7fb1c4009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.030+0000 7fb1cd7fa640 1 -- 172.21.15.161:0/3454964332 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb1bc0553d0 con 0x7fb1d00ac620 2024-03-23T14:36:01.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.266+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7fb1d004f550 con 0x7fb1d00ac620 2024-03-23T14:36:01.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.266+0000 7fb1cd7fa640 1 -- 172.21.15.161:0/3454964332 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v4) ==== 80+0+6645 (secure 0 0 0) 0x7fb1bc058a90 con 0x7fb1d00ac620 2024-03-23T14:36:01.262 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lzaKfFRAAc+l/nk//V/gpMXQ1JO3ATg==", 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.263 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lcMUxIhAAA7sdubbg/sFxR2JSLj6knA==", 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lOl8SHRAAPqXulseV9GlBMsoYeIUt5Q==", 2024-03-23T14:36:01.264 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5la0WTKRAACwJ5zIvawF/Gsu3pgRkjRQ==", 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:01.265 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBl6P5lv7pCExAACgyFS3tWsnorh+23JfVMFQ==", 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.266 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBo6P5lN9XjERAAY7Doq86QfXJ46LU3H9ET4g==", 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBr6P5le9jdDxAAF7YY5jy4+03N9tB79FqgxA==", 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:01.267 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBu6P5lfyVdDhAA6GSFuGvPYAUCczsnMpXFkQ==", 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQB06P5lxMy+FhAAhO1d5E1ioKlC3bqbkingpA==", 2024-03-23T14:36:01.268 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-23T14:36:01.269 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQB36P5lHl2nFRAA18JrgqJgj6mYykRjhemRvQ==", 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQB66P5lC5OzFBAAeJslkQ08ZxK2fx9yQKsCOA==", 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.270 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQB96P5lyiCSEhAAn+5LSA94oOhlckTfTUd7Lw==", 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g==", 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.271 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lskU4NxAAed0RZpsIuzbux+yi642hsw==", 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-23T14:36:01.272 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.2", 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lIYbHMxAAOYcn/3VZTDW2K3OBKfjxkA==", 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.273 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.3", 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBi6P5lwCRzARAAlFyun7fjX8hB/KsVJhLfWg==", 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-23T14:36:01.274 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBg6P5lHQT4MxAAvLp+7Sae/Py5kxSOJ51MzQ==", 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-mds", 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQCI6P5lakhaDxAASO5/aMAFZG+31rhG+li5Rw==", 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-mds" 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.275 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-mgr", 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQCI6P5lq1FaDxAAgI9oy60knwZQQ82agba3dA==", 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-mgr" 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-osd", 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQCI6P5lQVlaDxAAcEoUVz5ecuoL+8dzRRbRHw==", 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-osd" 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.276 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-rbd", 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQCI6P5l9WFaDxAAtcXejA1RMtO4QCi3/krrTg==", 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-rbd" 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-rbd-mirror", 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQCI6P5lxWhaDxAAtbn7JYkMNWLpP+nMPipbJw==", 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.277 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.bootstrap-rgw", 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQCI6P5lj29aDxAAn939iYjagimQBWRH4D1ioQ==", 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile bootstrap-rgw" 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lz/YnEhAA4Q/N4Xnmj7rkBGQH0WFU5w==", 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-23T14:36:01.278 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lMbfQDRAA7IKJtMZ6Z77M6m4dD75Vcg==", 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-23T14:36:01.279 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:36:01.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb1b406b810 msgr2=0x7fb1b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb1b406b810 0x7fb1b406dcd0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fb1c4034000 tx=0x7fb1c4009030 comp rx=0 tx=0).stop 2024-03-23T14:36:01.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1d00ac620 msgr2=0x7fb1d0159fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1d00ac620 0x7fb1d0159fa0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc0119a0 tx=0x7fb1bc011e60 comp rx=0 tx=0).stop 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 shutdown_connections 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb1b406b810 0x7fb1b406dcd0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1d015a4e0 0x7fb1d01581d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1d00ac620 0x7fb1d0159fa0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 --2- 172.21.15.161:0/3454964332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1c00039a0 0x7fb1d014da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.268+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 >> 172.21.15.161:0/3454964332 conn(0x7fb1d00b17f0 msgr2=0x7fb1d006d480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.269+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 shutdown_connections 2024-03-23T14:36:01.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.269+0000 7fb1d6d6c640 1 -- 172.21.15.161:0/3454964332 wait complete. 2024-03-23T14:36:01.283 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2024-03-23T14:36:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.389+0000 7f5b748fb640 1 Processor -- start 2024-03-23T14:36:01.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.389+0000 7f5b748fb640 1 -- start start 2024-03-23T14:36:01.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.389+0000 7f5b748fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.389+0000 7f5b748fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b700d2b40 0x7f5b700b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.389+0000 7f5b748fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b700d0460 con 0x7f5b700cf9d0 2024-03-23T14:36:01.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b748fb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5b700d1050 con 0x7f5b700d2b40 2024-03-23T14:36:01.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b748fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5b700d06b0 con 0x7f5b7015e400 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6ed9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b700d2b40 0x7f5b700b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6ed9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b700d2b40 0x7f5b700b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38076/0 (socket says 172.21.15.161:38076) 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6ed9d640 1 -- 172.21.15.161:0/2817276497 learned_addr learned my addr 172.21.15.161:0/2817276497 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6e59c640 1 --2- 172.21.15.161:0/2817276497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6ed9d640 1 -- 172.21.15.161:0/2817276497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 msgr2=0x7f5b700cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6ed9d640 1 --2- 172.21.15.161:0/2817276497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6ed9d640 1 -- 172.21.15.161:0/2817276497 >> v1:172.21.15.183:6789/0 conn(0x7f5b700cf9d0 legacy=0x7f5b700d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6ed9d640 1 -- 172.21.15.161:0/2817276497 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b700d0990 con 0x7f5b700d2b40 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.390+0000 7f5b6e59c640 1 --2- 172.21.15.161:0/2817276497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b6ed9d640 1 --2- 172.21.15.161:0/2817276497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b700b0a80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009e50 tx=0x7f5b6802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9c28e48210eb422 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b6dd9b640 1 -- 172.21.15.161:0/2817276497 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6802cb00 con 0x7f5b700d2b40 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b6dd9b640 1 -- 172.21.15.161:0/2817276497 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b6802cc60 con 0x7f5b700d2b40 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b6dd9b640 1 -- 172.21.15.161:0/2817276497 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b680343f0 con 0x7f5b700d2b40 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b748fb640 1 -- 172.21.15.161:0/2817276497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 msgr2=0x7f5b700b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b748fb640 1 --2- 172.21.15.161:0/2817276497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b700b0a80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f5b68009e50 tx=0x7f5b6802bf60 comp rx=0 tx=0).stop 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b748fb640 1 -- 172.21.15.161:0/2817276497 shutdown_connections 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b748fb640 1 --2- 172.21.15.161:0/2817276497 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b700b0a80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b748fb640 1 --2- 172.21.15.161:0/2817276497 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b748fb640 1 -- 172.21.15.161:0/2817276497 >> 172.21.15.161:0/2817276497 conn(0x7f5b7009d190 msgr2=0x7f5b700d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b748fb640 1 -- 172.21.15.161:0/2817276497 shutdown_connections 2024-03-23T14:36:01.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.391+0000 7f5b748fb640 1 -- 172.21.15.161:0/2817276497 wait complete. 2024-03-23T14:36:01.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.393+0000 7f5b748fb640 1 Processor -- start 2024-03-23T14:36:01.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.393+0000 7f5b748fb640 1 -- start start 2024-03-23T14:36:01.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b748fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b700cf9d0 0x7f5b701f7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b748fb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b701593d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b748fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700b8840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b748fb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5b701f4900 con 0x7f5b700d2b40 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b748fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5b701f4be0 con 0x7f5b700cf9d0 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b748fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5b701f7310 con 0x7f5b7015e400 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b701593d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b701593d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38086/0 (socket says 172.21.15.161:38086) 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 -- 172.21.15.161:0/1491529680 learned_addr learned my addr 172.21.15.161:0/1491529680 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b6e59c640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b700cf9d0 0x7f5b701f7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b6ed9d640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700b8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 -- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 msgr2=0x7f5b700b8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700b8840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 -- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b700cf9d0 msgr2=0x7f5b701f7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b700cf9d0 0x7f5b701f7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 -- 172.21.15.161:0/1491529680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b68009b30 con 0x7f5b700d2b40 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b6ed9d640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b65d9b640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b701593d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c400 tx=0x7f5b5800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b677fe640 1 -- 172.21.15.161:0/1491529680 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b580052d0 con 0x7f5b700d2b40 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b677fe640 1 -- 172.21.15.161:0/1491529680 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b58005450 con 0x7f5b700d2b40 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b701f75f0 con 0x7f5b700d2b40 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b677fe640 1 -- 172.21.15.161:0/1491529680 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b58007260 con 0x7f5b700d2b40 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.394+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b700ca780 con 0x7f5b700d2b40 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.395+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b701f7b10 con 0x7f5b700d2b40 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.396+0000 7f5b677fe640 1 -- 172.21.15.161:0/1491529680 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5b580073c0 con 0x7f5b700d2b40 2024-03-23T14:36:01.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.396+0000 7f5b677fe640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5006b810 0x7f5b5006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.396+0000 7f5b677fe640 1 -- 172.21.15.161:0/1491529680 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f5b580892a0 con 0x7f5b700d2b40 2024-03-23T14:36:01.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.396+0000 7f5b6e59c640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5006b810 0x7f5b5006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.397+0000 7f5b6e59c640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5006b810 0x7f5b5006dcd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5b60006fd0 tx=0x7f5b60008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.400+0000 7f5b677fe640 1 -- 172.21.15.161:0/1491529680 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5b701f7b10 con 0x7f5b700d2b40 2024-03-23T14:36:01.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.640+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) -- 0x7f5b701f7b10 con 0x7f5b700d2b40 2024-03-23T14:36:01.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.642+0000 7f5b677fe640 1 -- 172.21.15.161:0/1491529680 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 v5) ==== 90+0+0 (secure 0 0 0) 0x7f5b701f7b10 con 0x7f5b700d2b40 2024-03-23T14:36:01.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5006b810 msgr2=0x7f5b5006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5006b810 0x7f5b5006dcd0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f5b60006fd0 tx=0x7f5b60008040 comp rx=0 tx=0).stop 2024-03-23T14:36:01.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 msgr2=0x7f5b701593d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b701593d0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800c400 tx=0x7f5b5800c8c0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 shutdown_connections 2024-03-23T14:36:01.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5006b810 0x7f5b5006dcd0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b7015e400 0x7f5b700b8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.640 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d2b40 0x7f5b701593d0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 --2- 172.21.15.161:0/1491529680 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b700cf9d0 0x7f5b701f7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 >> 172.21.15.161:0/1491529680 conn(0x7f5b7009d190 msgr2=0x7f5b700b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:01.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 shutdown_connections 2024-03-23T14:36:01.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.644+0000 7f5b748fb640 1 -- 172.21.15.161:0/1491529680 wait complete. 2024-03-23T14:36:01.651 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2024-03-23T14:36:01.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.756+0000 7f92542ed640 1 Processor -- start 2024-03-23T14:36:01.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.756+0000 7f92542ed640 1 -- start start 2024-03-23T14:36:01.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f92542ed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c072100 0x7f924c15b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f92542ed640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f924c0724e0 0x7f924c06e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f92542ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f924c15be70 con 0x7f924c15b930 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f92542ed640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f924c15c430 con 0x7f924c0724e0 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f92542ed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f924c15c150 con 0x7f924c072100 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f925288a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f924c15b930 0x7f924c15e9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38906/0 (socket says 172.21.15.161:38906) 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f925288a640 1 -- 172.21.15.161:0/2664650752 learned_addr learned my addr 172.21.15.161:0/2664650752 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251888640 1 --2- 172.21.15.161:0/2664650752 >> v2:172.21.15.183:3300/0 conn(0x7f924c0724e0 0x7f924c06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9252089640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c072100 0x7f924c15b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3384051996 0 0) 0x7f924c15be70 con 0x7f924c15b930 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9238003520 con 0x7f924c15b930 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4013865073 0 0) 0x7f9238003520 con 0x7f924c15b930 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c072100 msgr2=0x7f924c15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c072100 0x7f924c15b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 >> v2:172.21.15.183:3300/0 conn(0x7f924c0724e0 msgr2=0x7f924c06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:01.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 --2- 172.21.15.161:0/2664650752 >> v2:172.21.15.183:3300/0 conn(0x7f924c0724e0 0x7f924c06e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c0b2c70 con 0x7f924c15b930 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f92440019f0 con 0x7f924c15b930 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 >> v1:172.21.15.161:6789/0 conn(0x7f924c15b930 legacy=0x7f924c15e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238003970 0x7f9238003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c072100 0x7f92380047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9238004d00 0x7f9238005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f924c15c430 con 0x7f9238004d00 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f924c15c150 con 0x7f9238003970 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.757+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f923800b980 con 0x7f924c072100 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f925288a640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9238004d00 0x7f9238005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f9252089640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238003970 0x7f9238003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f9251888640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c072100 0x7f92380047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f925288a640 1 -- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c072100 msgr2=0x7f92380047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f925288a640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c072100 0x7f92380047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f925288a640 1 -- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238003970 msgr2=0x7f9238003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f925288a640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238003970 0x7f9238003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f925288a640 1 -- 172.21.15.161:0/2664650752 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9238003520 con 0x7f9238004d00 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.758+0000 7f9252089640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238003970 0x7f9238003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f925288a640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9238004d00 0x7f9238005180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f924c15bda0 tx=0x7f92380057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2e67c26082b0fdd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9244034630 con 0x7f9238004d00 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f924402f070 con 0x7f9238004d00 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f9251087640 1 -- 172.21.15.161:0/2664650752 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92440349d0 con 0x7f9238004d00 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f92542ed640 1 -- 172.21.15.161:0/2664650752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9238004d00 msgr2=0x7f9238005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f92542ed640 1 --2- 172.21.15.161:0/2664650752 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9238004d00 0x7f9238005180 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f924c15bda0 tx=0x7f92380057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f925288a640 1 -- 172.21.15.161:0/2664650752 reap_dead start 2024-03-23T14:36:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f92542ed640 1 -- 172.21.15.161:0/2664650752 shutdown_connections 2024-03-23T14:36:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f92542ed640 1 -- 172.21.15.161:0/2664650752 >> 172.21.15.161:0/2664650752 conn(0x7f924c0b17f0 msgr2=0x7f924c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f92542ed640 1 -- 172.21.15.161:0/2664650752 shutdown_connections 2024-03-23T14:36:01.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.759+0000 7f92542ed640 1 -- 172.21.15.161:0/2664650752 wait complete. 2024-03-23T14:36:01.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.760+0000 7f92542ed640 1 Processor -- start 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f92542ed640 1 -- start start 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f92542ed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238004d00 0x7f924c14b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f92542ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f924c151fe0 0x7f924c1504f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f92542ed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c150a30 0x7f924c06d480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f92542ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f924c150f40 con 0x7f924c151fe0 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f92542ed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f924c151220 con 0x7f9238004d00 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f92542ed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f924c151500 con 0x7f924c150a30 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f9251888640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f924c151fe0 0x7f924c1504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f9251888640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f924c151fe0 0x7f924c1504f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38096/0 (socket says 172.21.15.161:38096) 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.761+0000 7f9251888640 1 -- 172.21.15.161:0/1054386427 learned_addr learned my addr 172.21.15.161:0/1054386427 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f925288a640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c150a30 0x7f924c06d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9252089640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238004d00 0x7f924c14b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9251888640 1 -- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c150a30 msgr2=0x7f924c06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9251888640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c150a30 0x7f924c06d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9251888640 1 -- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238004d00 msgr2=0x7f924c14b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9251888640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238004d00 0x7f924c14b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9251888640 1 -- 172.21.15.161:0/1054386427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f924c1517e0 con 0x7f924c151fe0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9252089640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238004d00 0x7f924c14b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9251888640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f924c151fe0 0x7f924c1504f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f923c011a70 tx=0x7f923c011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9242ffd640 1 -- 172.21.15.161:0/1054386427 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c009680 con 0x7f924c151fe0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9242ffd640 1 -- 172.21.15.161:0/1054386427 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f923c00fb50 con 0x7f924c151fe0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f924c151ac0 con 0x7f924c151fe0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f9242ffd640 1 -- 172.21.15.161:0/1054386427 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f923c00fee0 con 0x7f924c151fe0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.762+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f924c155c70 con 0x7f924c151fe0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.763+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f924c15b6f0 con 0x7f924c151fe0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.763+0000 7f9242ffd640 1 -- 172.21.15.161:0/1054386427 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f923c014440 con 0x7f924c151fe0 2024-03-23T14:36:01.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.764+0000 7f9242ffd640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f923006b840 0x7f923006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.764+0000 7f9242ffd640 1 -- 172.21.15.161:0/1054386427 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f923c08a7c0 con 0x7f924c151fe0 2024-03-23T14:36:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.764+0000 7f9252089640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f923006b840 0x7f923006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.764+0000 7f9252089640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f923006b840 0x7f923006dd00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9248006b10 tx=0x7f9248008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:01.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:01.768+0000 7f9242ffd640 1 -- 172.21.15.161:0/1054386427 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f923c055530 con 0x7f924c151fe0 2024-03-23T14:36:02.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.010+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) -- 0x7f924c0735e0 con 0x7f924c151fe0 2024-03-23T14:36:02.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.012+0000 7f9242ffd640 1 -- 172.21.15.161:0/1054386427 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 v6) ==== 90+0+0 (secure 0 0 0) 0x7f923c058bf0 con 0x7f924c151fe0 2024-03-23T14:36:02.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f923006b840 msgr2=0x7f923006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f923006b840 0x7f923006dd00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f9248006b10 tx=0x7f9248008040 comp rx=0 tx=0).stop 2024-03-23T14:36:02.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f924c151fe0 msgr2=0x7f924c1504f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f924c151fe0 0x7f924c1504f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f923c011a70 tx=0x7f923c011f30 comp rx=0 tx=0).stop 2024-03-23T14:36:02.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 shutdown_connections 2024-03-23T14:36:02.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f923006b840 0x7f923006dd00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f924c150a30 0x7f924c06d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f924c151fe0 0x7f924c1504f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 --2- 172.21.15.161:0/1054386427 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9238004d00 0x7f924c14b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 >> 172.21.15.161:0/1054386427 conn(0x7f924c0b17f0 msgr2=0x7f924c15d400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:02.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 shutdown_connections 2024-03-23T14:36:02.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.014+0000 7f92542ed640 1 -- 172.21.15.161:0/1054386427 wait complete. 2024-03-23T14:36:02.021 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2024-03-23T14:36:02.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.127+0000 7f511068f640 1 Processor -- start 2024-03-23T14:36:02.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.127+0000 7f511068f640 1 -- start start 2024-03-23T14:36:02.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f511068f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f510806eca0 0x7f5108165d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f511068f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f510815be70 con 0x7f51080724e0 2024-03-23T14:36:02.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f511068f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f510815c150 con 0x7f5108072100 2024-03-23T14:36:02.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f511068f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f510815c430 con 0x7f510806eca0 2024-03-23T14:36:02.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f510ec2c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f510806eca0 0x7f5108165d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f510ec2c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f510806eca0 0x7f5108165d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38098/0 (socket says 172.21.15.161:38098) 2024-03-23T14:36:02.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f510ec2c640 1 -- 172.21.15.161:0/176737232 learned_addr learned my addr 172.21.15.161:0/176737232 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:02.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f510ec2c640 1 -- 172.21.15.161:0/176737232 >> v1:172.21.15.183:6789/0 conn(0x7f5108072100 legacy=0x7f510815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:02.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f510ec2c640 1 -- 172.21.15.161:0/176737232 >> v1:172.21.15.161:6789/0 conn(0x7f51080724e0 legacy=0x7f510806e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f510d429640 1 -- 172.21.15.161:0/176737232 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326475733 0 0) 0x7f510815be70 con 0x7f51080724e0 2024-03-23T14:36:02.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.128+0000 7f510ec2c640 1 -- 172.21.15.161:0/176737232 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51080b2c70 con 0x7f510806eca0 2024-03-23T14:36:02.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.129+0000 7f510ec2c640 1 --2- 172.21.15.161:0/176737232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f510806eca0 0x7f5108165d70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5100009e50 tx=0x7f510002bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebed310112a7716d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.129+0000 7f510d429640 1 -- 172.21.15.161:0/176737232 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f510002cb20 con 0x7f510806eca0 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.129+0000 7f510d429640 1 -- 172.21.15.161:0/176737232 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f510002cc80 con 0x7f510806eca0 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.129+0000 7f510d429640 1 -- 172.21.15.161:0/176737232 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51000343f0 con 0x7f510806eca0 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.130+0000 7f511068f640 1 -- 172.21.15.161:0/176737232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f510806eca0 msgr2=0x7f5108165d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.130+0000 7f511068f640 1 --2- 172.21.15.161:0/176737232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f510806eca0 0x7f5108165d70 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f5100009e50 tx=0x7f510002bf80 comp rx=0 tx=0).stop 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.130+0000 7f511068f640 1 -- 172.21.15.161:0/176737232 shutdown_connections 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.130+0000 7f511068f640 1 --2- 172.21.15.161:0/176737232 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f510806eca0 0x7f5108165d70 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.130+0000 7f511068f640 1 -- 172.21.15.161:0/176737232 >> 172.21.15.161:0/176737232 conn(0x7f51080b17f0 msgr2=0x7f51080b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.130+0000 7f511068f640 1 -- 172.21.15.161:0/176737232 shutdown_connections 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.130+0000 7f511068f640 1 -- 172.21.15.161:0/176737232 wait complete. 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.131+0000 7f511068f640 1 Processor -- start 2024-03-23T14:36:02.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.131+0000 7f511068f640 1 -- start start 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f511068f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f510806eca0 0x7f51081426e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f511068f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5108072100 0x7f51080a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f511068f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51080724e0 0x7f51080a99e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f511068f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51081544c0 con 0x7f5108072100 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f511068f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5108145cd0 con 0x7f51080724e0 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f511068f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5108145fb0 con 0x7f510806eca0 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5108072100 0x7f51080a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5108072100 0x7f51080a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38102/0 (socket says 172.21.15.161:38102) 2024-03-23T14:36:02.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 -- 172.21.15.161:0/1707546940 learned_addr learned my addr 172.21.15.161:0/1707546940 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510ec2c640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51080724e0 0x7f51080a99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510e42b640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f510806eca0 0x7f51081426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 -- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f510806eca0 msgr2=0x7f51081426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f510806eca0 0x7f51081426e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 -- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51080724e0 msgr2=0x7f51080a99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51080724e0 0x7f51080a99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 -- 172.21.15.161:0/1707546940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5100009b30 con 0x7f5108072100 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510ec2c640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51080724e0 0x7f51080a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f510dc2a640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5108072100 0x7f51080a94a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f50f800c410 tx=0x7f50f800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f50feffd640 1 -- 172.21.15.161:0/1707546940 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f80052d0 con 0x7f5108072100 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f50feffd640 1 -- 172.21.15.161:0/1707546940 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f50f8005430 con 0x7f5108072100 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51081430d0 con 0x7f5108072100 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f50feffd640 1 -- 172.21.15.161:0/1707546940 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f50f8003850 con 0x7f5108072100 2024-03-23T14:36:02.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.132+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5108143690 con 0x7f5108072100 2024-03-23T14:36:02.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.133+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51081438b0 con 0x7f5108072100 2024-03-23T14:36:02.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.134+0000 7f50feffd640 1 -- 172.21.15.161:0/1707546940 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f50f80039b0 con 0x7f5108072100 2024-03-23T14:36:02.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.134+0000 7f50feffd640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50ec06b810 0x7f50ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.134+0000 7f50feffd640 1 -- 172.21.15.161:0/1707546940 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f50f80894d0 con 0x7f5108072100 2024-03-23T14:36:02.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.134+0000 7f510e42b640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50ec06b810 0x7f50ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.135+0000 7f510e42b640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50ec06b810 0x7f50ec06dcd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5104000f60 tx=0x7f5104009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.138+0000 7f50feffd640 1 -- 172.21.15.161:0/1707546940 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f50f8054240 con 0x7f5108072100 2024-03-23T14:36:02.369 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.373+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) -- 0x7f51080735e0 con 0x7f5108072100 2024-03-23T14:36:02.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.375+0000 7f50feffd640 1 -- 172.21.15.161:0/1707546940 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 v7) ==== 90+0+0 (secure 0 0 0) 0x7f50f80170a0 con 0x7f5108072100 2024-03-23T14:36:02.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50ec06b810 msgr2=0x7f50ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50ec06b810 0x7f50ec06dcd0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5104000f60 tx=0x7f5104009040 comp rx=0 tx=0).stop 2024-03-23T14:36:02.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5108072100 msgr2=0x7f51080a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5108072100 0x7f51080a94a0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f50f800c410 tx=0x7f50f800c8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 shutdown_connections 2024-03-23T14:36:02.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50ec06b810 0x7f50ec06dcd0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51080724e0 0x7f51080a99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5108072100 0x7f51080a94a0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 --2- 172.21.15.161:0/1707546940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f510806eca0 0x7f51081426e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 >> 172.21.15.161:0/1707546940 conn(0x7f51080b17f0 msgr2=0x7f51080d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:02.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 shutdown_connections 2024-03-23T14:36:02.375 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.377+0000 7f511068f640 1 -- 172.21.15.161:0/1707546940 wait complete. 2024-03-23T14:36:02.383 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2024-03-23T14:36:02.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.489+0000 7f52b691e640 1 Processor -- start 2024-03-23T14:36:02.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.489+0000 7f52b691e640 1 -- start start 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52b691e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b006eb30 0x7f52b00b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52b691e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f52b00b4260 0x7f52b015b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52b691e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52b0070b90 con 0x7f52b00b3610 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52b691e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f52b015ab90 con 0x7f52b00b4260 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52b691e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f52b00713a0 con 0x7f52b006eb30 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52b4ebb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f52b00b3610 0x7f52b006d480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38926/0 (socket says 172.21.15.161:38926) 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52b4ebb640 1 -- 172.21.15.161:0/3659496274 learned_addr learned my addr 172.21.15.161:0/3659496274 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52af7fe640 1 --2- 172.21.15.161:0/3659496274 >> v2:172.21.15.183:3300/0 conn(0x7f52b00b4260 0x7f52b015b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52affff640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b006eb30 0x7f52b00b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 683386633 0 0) 0x7f52b0070b90 con 0x7f52b00b3610 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5298003520 con 0x7f52b00b3610 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1025251916 0 0) 0x7f5298003520 con 0x7f52b00b3610 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b006eb30 msgr2=0x7f52b00b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b006eb30 0x7f52b00b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 >> v2:172.21.15.183:3300/0 conn(0x7f52b00b4260 msgr2=0x7f52b015b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 --2- 172.21.15.161:0/3659496274 >> v2:172.21.15.183:3300/0 conn(0x7f52b00b4260 0x7f52b015b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b015b0d0 con 0x7f52b00b3610 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f52a80019f0 con 0x7f52b00b3610 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 >> v1:172.21.15.161:6789/0 conn(0x7f52b00b3610 legacy=0x7f52b006d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b006eb30 0x7f5298003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5298003ef0 0x7f52980047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5298004d00 0x7f5298005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f52b015ab90 con 0x7f5298004d00 2024-03-23T14:36:02.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f52b00713a0 con 0x7f5298003ef0 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f529800b980 con 0x7f52b006eb30 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52b4ebb640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5298004d00 0x7f5298005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52af7fe640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5298003ef0 0x7f52980047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.490+0000 7f52affff640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b006eb30 0x7f5298003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.491+0000 7f52b4ebb640 1 -- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b006eb30 msgr2=0x7f5298003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.491+0000 7f52b4ebb640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b006eb30 0x7f5298003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.491+0000 7f52b4ebb640 1 -- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5298003ef0 msgr2=0x7f52980047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.491+0000 7f52b4ebb640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5298003ef0 0x7f52980047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.491+0000 7f52b4ebb640 1 -- 172.21.15.161:0/3659496274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5298003520 con 0x7f5298004d00 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.491+0000 7f52b4ebb640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5298004d00 0x7f5298005180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f52a800a8d0 tx=0x7f52980057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a4f2af545de160aa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.491+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a8034630 con 0x7f5298004d00 2024-03-23T14:36:02.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52a802f070 con 0x7f5298004d00 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52aeffd640 1 -- 172.21.15.161:0/3659496274 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a80349d0 con 0x7f5298004d00 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52b691e640 1 -- 172.21.15.161:0/3659496274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5298004d00 msgr2=0x7f5298005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52b691e640 1 --2- 172.21.15.161:0/3659496274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5298004d00 0x7f5298005180 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f52a800a8d0 tx=0x7f52980057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52b4ebb640 1 -- 172.21.15.161:0/3659496274 reap_dead start 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52b691e640 1 -- 172.21.15.161:0/3659496274 shutdown_connections 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52b691e640 1 -- 172.21.15.161:0/3659496274 >> 172.21.15.161:0/3659496274 conn(0x7f52b00b17f0 msgr2=0x7f52b00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52b691e640 1 -- 172.21.15.161:0/3659496274 shutdown_connections 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.492+0000 7f52b691e640 1 -- 172.21.15.161:0/3659496274 wait complete. 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.493+0000 7f52b691e640 1 Processor -- start 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.493+0000 7f52b691e640 1 -- start start 2024-03-23T14:36:02.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52b691e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52b013cad0 0x7f52b00cd570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52b691e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52b013ceb0 0x7f52b00cee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52b691e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b01463c0 0x7f52b0166520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52b691e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f52b01419f0 con 0x7f52b013cad0 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52b691e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f52b00cc840 con 0x7f52b013ceb0 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52b691e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f52b01468f0 con 0x7f52b01463c0 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52affff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52b013cad0 0x7f52b00cd570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52affff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52b013cad0 0x7f52b00cd570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38120/0 (socket says 172.21.15.161:38120) 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.494+0000 7f52affff640 1 -- 172.21.15.161:0/1928561312 learned_addr learned my addr 172.21.15.161:0/1928561312 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52affff640 1 -- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b01463c0 msgr2=0x7f52b0166520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52affff640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b01463c0 0x7f52b0166520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52affff640 1 -- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52b013ceb0 msgr2=0x7f52b00cee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52affff640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52b013ceb0 0x7f52b00cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52affff640 1 -- 172.21.15.161:0/1928561312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52b00cca90 con 0x7f52b013cad0 2024-03-23T14:36:02.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52affff640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52b013cad0 0x7f52b00cd570 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f52a00119a0 tx=0x7f52a0011e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52acff9640 1 -- 172.21.15.161:0/1928561312 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a000fa70 con 0x7f52b013cad0 2024-03-23T14:36:02.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52acff9640 1 -- 172.21.15.161:0/1928561312 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52a000fbd0 con 0x7f52b013cad0 2024-03-23T14:36:02.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52b00ccd70 con 0x7f52b013cad0 2024-03-23T14:36:02.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52acff9640 1 -- 172.21.15.161:0/1928561312 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52a0014440 con 0x7f52b013cad0 2024-03-23T14:36:02.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.495+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52b00cdab0 con 0x7f52b013cad0 2024-03-23T14:36:02.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.496+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52b004f550 con 0x7f52b013cad0 2024-03-23T14:36:02.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.496+0000 7f52acff9640 1 -- 172.21.15.161:0/1928561312 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f52a00145a0 con 0x7f52b013cad0 2024-03-23T14:36:02.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.497+0000 7f52acff9640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f529c06b810 0x7f529c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.497+0000 7f52acff9640 1 -- 172.21.15.161:0/1928561312 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f52a008a620 con 0x7f52b013cad0 2024-03-23T14:36:02.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.497+0000 7f52af7fe640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f529c06b810 0x7f529c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.497+0000 7f52af7fe640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f529c06b810 0x7f529c06dcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f52a4006e80 tx=0x7f52a4006df0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.501+0000 7f52acff9640 1 -- 172.21.15.161:0/1928561312 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f52a0055420 con 0x7f52b013cad0 2024-03-23T14:36:02.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.738+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) -- 0x7f52b00b2ee0 con 0x7f52b013cad0 2024-03-23T14:36:02.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.741+0000 7f52acff9640 1 -- 172.21.15.161:0/1928561312 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 v8) ==== 90+0+0 (secure 0 0 0) 0x7f52a0058ae0 con 0x7f52b013cad0 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f529c06b810 msgr2=0x7f529c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f529c06b810 0x7f529c06dcd0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f52a4006e80 tx=0x7f52a4006df0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52b013cad0 msgr2=0x7f52b00cd570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52b013cad0 0x7f52b00cd570 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f52a00119a0 tx=0x7f52a0011e60 comp rx=0 tx=0).stop 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 shutdown_connections 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f529c06b810 0x7f529c06dcd0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f52b01463c0 0x7f52b0166520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f52b013ceb0 0x7f52b00cee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 --2- 172.21.15.161:0/1928561312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f52b013cad0 0x7f52b00cd570 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 >> 172.21.15.161:0/1928561312 conn(0x7f52b00b17f0 msgr2=0x7f52b015e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:02.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 shutdown_connections 2024-03-23T14:36:02.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.743+0000 7f52b691e640 1 -- 172.21.15.161:0/1928561312 wait complete. 2024-03-23T14:36:02.749 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2024-03-23T14:36:02.850 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.854+0000 7fc80f0ce640 1 Processor -- start 2024-03-23T14:36:02.851 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.855+0000 7fc80f0ce640 1 -- start start 2024-03-23T14:36:02.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.855+0000 7fc80f0ce640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc8080724e0 0x7fc80806e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.855+0000 7fc80f0ce640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80815be70 con 0x7fc80806ead0 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.855+0000 7fc80f0ce640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc80815c150 con 0x7fc808072100 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.855+0000 7fc80f0ce640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc80806ef80 con 0x7fc8080724e0 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc80d66b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc80806ead0 0x7fc8080b3470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38942/0 (socket says 172.21.15.161:38942) 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc80d66b640 1 -- 172.21.15.161:0/1988859256 learned_addr learned my addr 172.21.15.161:0/1988859256 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc807fff640 1 --2- 172.21.15.161:0/1988859256 >> v2:172.21.15.183:3300/0 conn(0x7fc8080724e0 0x7fc80806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636495376 0 0) 0x7fc80815be70 con 0x7fc80806ead0 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7f8003520 con 0x7fc80806ead0 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3387436976 0 0) 0x7fc7f8003520 con 0x7fc80806ead0 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 >> v2:172.21.15.183:3300/0 conn(0x7fc8080724e0 msgr2=0x7fc80806e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 --2- 172.21.15.161:0/1988859256 >> v2:172.21.15.183:3300/0 conn(0x7fc8080724e0 0x7fc80806e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 >> v1:172.21.15.183:6789/0 conn(0x7fc808072100 legacy=0x7fc80815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8080b2c70 con 0x7fc80806ead0 2024-03-23T14:36:02.855 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fc8000019f0 con 0x7fc80806ead0 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 >> v1:172.21.15.161:6789/0 conn(0x7fc80806ead0 legacy=0x7fc8080b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f8003970 0x7fc7f8003d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc7f8004840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f8005040 0x7fc7f800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f800bf70 con 0x7fc7f8004370 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc7f800c0e0 con 0x7fc7f8005040 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc7f800c2a0 con 0x7fc7f8003970 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc807fff640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc7f8004840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc80d66b640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f8005040 0x7fc7f800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc80ce6a640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f8003970 0x7fc7f8003d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc807fff640 1 -- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f8003970 msgr2=0x7fc7f8003d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc807fff640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc7f8003970 0x7fc7f8003d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc807fff640 1 -- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f8005040 msgr2=0x7fc7f800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:02.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc807fff640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f8005040 0x7fc7f800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc807fff640 1 -- 172.21.15.161:0/1988859256 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7f8003520 con 0x7fc7f8004370 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.856+0000 7fc80d66b640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc7f8005040 0x7fc7f800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.857+0000 7fc80d66b640 1 -- 172.21.15.161:0/1988859256 reap_dead start 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.857+0000 7fc807fff640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc7f8004840 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009a90 tx=0x7fc7f80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc71488630df3cfd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.857+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f402c680 con 0x7fc7f8004370 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.857+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7f402c7e0 con 0x7fc7f8004370 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.857+0000 7fc8077fe640 1 -- 172.21.15.161:0/1988859256 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7f402cb70 con 0x7fc7f8004370 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.858+0000 7fc80f0ce640 1 -- 172.21.15.161:0/1988859256 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 msgr2=0x7fc7f8004840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.858+0000 7fc80f0ce640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc7f8004840 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4009a90 tx=0x7fc7f80057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.858+0000 7fc80f0ce640 1 -- 172.21.15.161:0/1988859256 shutdown_connections 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.858+0000 7fc80f0ce640 1 --2- 172.21.15.161:0/1988859256 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc7f8004840 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.858+0000 7fc80f0ce640 1 -- 172.21.15.161:0/1988859256 >> 172.21.15.161:0/1988859256 conn(0x7fc8080b17f0 msgr2=0x7fc8080b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.858+0000 7fc80f0ce640 1 -- 172.21.15.161:0/1988859256 shutdown_connections 2024-03-23T14:36:02.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.858+0000 7fc80f0ce640 1 -- 172.21.15.161:0/1988859256 wait complete. 2024-03-23T14:36:02.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.859+0000 7fc80f0ce640 1 Processor -- start 2024-03-23T14:36:02.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.859+0000 7fc80f0ce640 1 -- start start 2024-03-23T14:36:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80f0ce640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc8081f15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80f0ce640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc80815a740 0x7fc808150140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80f0ce640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc80814ef60 0x7fc8080b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80f0ce640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc808150680 con 0x7fc7f8004370 2024-03-23T14:36:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80f0ce640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc8081576e0 con 0x7fc80814ef60 2024-03-23T14:36:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80f0ce640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc8081508b0 con 0x7fc80815a740 2024-03-23T14:36:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc8081f15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc8081f15e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38138/0 (socket says 172.21.15.161:38138) 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 -- 172.21.15.161:0/3047802053 learned_addr learned my addr 172.21.15.161:0/3047802053 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 -- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc80815a740 msgr2=0x7fc808150140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc80815a740 0x7fc808150140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 -- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc80814ef60 msgr2=0x7fc8080b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc80814ef60 0x7fc8080b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 -- 172.21.15.161:0/3047802053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc808157930 con 0x7fc7f8004370 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.860+0000 7fc80ce6a640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc8081f15e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00e7d0 tx=0x7fc7fc00ec90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.861+0000 7fc8057fa640 1 -- 172.21.15.161:0/3047802053 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc0050d0 con 0x7fc7f8004370 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.861+0000 7fc8057fa640 1 -- 172.21.15.161:0/3047802053 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7fc005230 con 0x7fc7f8004370 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.861+0000 7fc8057fa640 1 -- 172.21.15.161:0/3047802053 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7fc00c640 con 0x7fc7f8004370 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.861+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc808157c10 con 0x7fc7f8004370 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.861+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc80815abc0 con 0x7fc7f8004370 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.862+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc808158130 con 0x7fc7f8004370 2024-03-23T14:36:02.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.862+0000 7fc8057fa640 1 -- 172.21.15.161:0/3047802053 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc7fc00c7a0 con 0x7fc7f8004370 2024-03-23T14:36:02.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.862+0000 7fc8057fa640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc7ec06b840 0x7fc7ec06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:02.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.862+0000 7fc8057fa640 1 -- 172.21.15.161:0/3047802053 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fc7fc089c00 con 0x7fc7f8004370 2024-03-23T14:36:02.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.862+0000 7fc807fff640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc7ec06b840 0x7fc7ec06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:02.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.863+0000 7fc807fff640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc7ec06b840 0x7fc7ec06dd00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4036000 tx=0x7fc7f402e000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:02.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:02.867+0000 7fc8057fa640 1 -- 172.21.15.161:0/3047802053 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc808158130 con 0x7fc7f8004370 2024-03-23T14:36:03.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.106+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) -- 0x7fc808158130 con 0x7fc7f8004370 2024-03-23T14:36:03.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.108+0000 7fc8057fa640 1 -- 172.21.15.161:0/3047802053 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 v9) ==== 97+0+0 (secure 0 0 0) 0x7fc808158130 con 0x7fc7f8004370 2024-03-23T14:36:03.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.110+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc7ec06b840 msgr2=0x7fc7ec06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.110+0000 7fc80f0ce640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc7ec06b840 0x7fc7ec06dd00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fc7f4036000 tx=0x7fc7f402e000 comp rx=0 tx=0).stop 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.110+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 msgr2=0x7fc8081f15e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.110+0000 7fc80f0ce640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc8081f15e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fc7fc00e7d0 tx=0x7fc7fc00ec90 comp rx=0 tx=0).stop 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.111+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 shutdown_connections 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.111+0000 7fc80f0ce640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc7ec06b840 0x7fc7ec06dd00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.111+0000 7fc80f0ce640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc80814ef60 0x7fc8080b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.111+0000 7fc80f0ce640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc80815a740 0x7fc808150140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.111+0000 7fc80f0ce640 1 --2- 172.21.15.161:0/3047802053 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7f8004370 0x7fc8081f15e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.111+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 >> 172.21.15.161:0/3047802053 conn(0x7fc8080b17f0 msgr2=0x7fc8080b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.111+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 shutdown_connections 2024-03-23T14:36:03.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.111+0000 7fc80f0ce640 1 -- 172.21.15.161:0/3047802053 wait complete. 2024-03-23T14:36:03.117 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2024-03-23T14:36:03.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.223+0000 7fa482084640 1 Processor -- start 2024-03-23T14:36:03.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.223+0000 7fa482084640 1 -- start start 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.223+0000 7fa482084640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c145080 0x7fa47c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.223+0000 7fa482084640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa47c1601f0 0x7fa47c15c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.223+0000 7fa482084640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa47c0cf4f0 con 0x7fa47c1607d0 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.223+0000 7fa482084640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa47c071010 con 0x7fa47c1601f0 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.223+0000 7fa482084640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa47c0cef30 con 0x7fa47c145080 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa47c1601f0 0x7fa47c15c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa47c1607d0 0x7fa47c15fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38952/0 (socket says 172.21.15.161:38952) 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47affd640 1 -- 172.21.15.161:0/3412045332 learned_addr learned my addr 172.21.15.161:0/3412045332 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47b7fe640 1 --2- 172.21.15.161:0/3412045332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c145080 0x7fa47c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47bfff640 1 -- 172.21.15.161:0/3412045332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c145080 msgr2=0x7fa47c072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:03.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47bfff640 1 --2- 172.21.15.161:0/3412045332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c145080 0x7fa47c072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47bfff640 1 -- 172.21.15.161:0/3412045332 >> v1:172.21.15.161:6789/0 conn(0x7fa47c1607d0 legacy=0x7fa47c15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47a7fc640 1 -- 172.21.15.161:0/3412045332 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3483116045 0 0) 0x7fa47c0cf4f0 con 0x7fa47c1607d0 2024-03-23T14:36:03.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.224+0000 7fa47bfff640 1 -- 172.21.15.161:0/3412045332 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa47c0cf210 con 0x7fa47c1601f0 2024-03-23T14:36:03.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa47bfff640 1 --2- 172.21.15.161:0/3412045332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c15c4f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa474009d90 tx=0x7fa47402be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65050208c37e9fd2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa47a7fc640 1 -- 172.21.15.161:0/3412045332 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa47402cbc0 con 0x7fa47c1601f0 2024-03-23T14:36:03.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa47a7fc640 1 -- 172.21.15.161:0/3412045332 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa47402cd20 con 0x7fa47c1601f0 2024-03-23T14:36:03.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa47a7fc640 1 -- 172.21.15.161:0/3412045332 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa474034660 con 0x7fa47c1601f0 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa482084640 1 -- 172.21.15.161:0/3412045332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 msgr2=0x7fa47c15c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa482084640 1 --2- 172.21.15.161:0/3412045332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c15c4f0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fa474009d90 tx=0x7fa47402be10 comp rx=0 tx=0).stop 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa482084640 1 -- 172.21.15.161:0/3412045332 shutdown_connections 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa482084640 1 --2- 172.21.15.161:0/3412045332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c15c4f0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa482084640 1 --2- 172.21.15.161:0/3412045332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c145080 0x7fa47c072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa482084640 1 -- 172.21.15.161:0/3412045332 >> 172.21.15.161:0/3412045332 conn(0x7fa47c0b17f0 msgr2=0x7fa47c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa482084640 1 -- 172.21.15.161:0/3412045332 shutdown_connections 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.225+0000 7fa482084640 1 -- 172.21.15.161:0/3412045332 wait complete. 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.226+0000 7fa482084640 1 Processor -- start 2024-03-23T14:36:03.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa482084640 1 -- start start 2024-03-23T14:36:03.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa482084640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa47c145080 0x7fa47c13d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa482084640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c0715a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa482084640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c1607d0 0x7fa47c15a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa482084640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c140050 con 0x7fa47c1601f0 2024-03-23T14:36:03.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa482084640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa47c15c230 con 0x7fa47c145080 2024-03-23T14:36:03.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa482084640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa47c13f0f0 con 0x7fa47c1607d0 2024-03-23T14:36:03.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa47affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c0715a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.227+0000 7fa47affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c0715a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38154/0 (socket says 172.21.15.161:38154) 2024-03-23T14:36:03.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47affd640 1 -- 172.21.15.161:0/253962659 learned_addr learned my addr 172.21.15.161:0/253962659 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:03.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47b7fe640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa47c145080 0x7fa47c13d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47bfff640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c1607d0 0x7fa47c15a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47affd640 1 -- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c1607d0 msgr2=0x7fa47c15a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47affd640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c1607d0 0x7fa47c15a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47affd640 1 -- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa47c145080 msgr2=0x7fa47c13d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47affd640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa47c145080 0x7fa47c13d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47affd640 1 -- 172.21.15.161:0/253962659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa474009a70 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa47affd640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c0715a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa47000c410 tx=0x7fa47000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa463fff640 1 -- 172.21.15.161:0/253962659 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4700074e0 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa463fff640 1 -- 172.21.15.161:0/253962659 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa470007660 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa463fff640 1 -- 172.21.15.161:0/253962659 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa470004a20 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa47c13f3d0 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.228+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa47c13f990 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.229+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa47c140190 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.229+0000 7fa463fff640 1 -- 172.21.15.161:0/253962659 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa470004b80 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.230+0000 7fa463fff640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa46806b810 0x7fa46806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.230+0000 7fa463fff640 1 -- 172.21.15.161:0/253962659 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fa470089ab0 con 0x7fa47c1601f0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.230+0000 7fa47b7fe640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa46806b810 0x7fa46806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.230+0000 7fa47b7fe640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa46806b810 0x7fa46806dcd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa46c006fd0 tx=0x7fa46c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.234+0000 7fa463fff640 1 -- 172.21.15.161:0/253962659 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa47c140190 con 0x7fa47c1601f0 2024-03-23T14:36:03.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.471+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) -- 0x7fa47c140190 con 0x7fa47c1601f0 2024-03-23T14:36:03.469 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.473+0000 7fa463fff640 1 -- 172.21.15.161:0/253962659 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 v10) ==== 90+0+0 (secure 0 0 0) 0x7fa47c140190 con 0x7fa47c1601f0 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa46806b810 msgr2=0x7fa46806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa46806b810 0x7fa46806dcd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa46c006fd0 tx=0x7fa46c008040 comp rx=0 tx=0).stop 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 msgr2=0x7fa47c0715a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c0715a0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa47000c410 tx=0x7fa47000c8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 shutdown_connections 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa46806b810 0x7fa46806dcd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa47c1607d0 0x7fa47c15a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa47c1601f0 0x7fa47c0715a0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 --2- 172.21.15.161:0/253962659 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa47c145080 0x7fa47c13d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.475+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 >> 172.21.15.161:0/253962659 conn(0x7fa47c0b17f0 msgr2=0x7fa47c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.476+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 shutdown_connections 2024-03-23T14:36:03.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.476+0000 7fa482084640 1 -- 172.21.15.161:0/253962659 wait complete. 2024-03-23T14:36:03.482 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:03.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.587+0000 7fac19c07640 1 Processor -- start 2024-03-23T14:36:03.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.587+0000 7fac19c07640 1 -- start start 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.588+0000 7fac19c07640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fac14145080 0x7fac14072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.588+0000 7fac19c07640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1415fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.588+0000 7fac19c07640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fac140cf4f0 con 0x7fac1406cbf0 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.588+0000 7fac19c07640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fac14071010 con 0x7fac14145080 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.588+0000 7fac19c07640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fac140cef30 con 0x7fac141607d0 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac12ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1415fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac137fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fac14145080 0x7fac14072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac13fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fac1406cbf0 0x7fac14071430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:44126/0 (socket says 172.21.15.161:44126) 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac13fff640 1 -- 172.21.15.161:0/2587470833 learned_addr learned my addr 172.21.15.161:0/2587470833 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac137fe640 1 -- 172.21.15.161:0/2587470833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 msgr2=0x7fac1415fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac137fe640 1 --2- 172.21.15.161:0/2587470833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1415fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac137fe640 1 -- 172.21.15.161:0/2587470833 >> v1:172.21.15.183:6789/0 conn(0x7fac1406cbf0 legacy=0x7fac14071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac137fe640 1 -- 172.21.15.161:0/2587470833 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac140cf210 con 0x7fac14145080 2024-03-23T14:36:03.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac127fc640 1 -- 172.21.15.161:0/2587470833 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2162095046 0 0) 0x7fac140cf4f0 con 0x7fac1406cbf0 2024-03-23T14:36:03.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.589+0000 7fac12ffd640 1 --2- 172.21.15.161:0/2587470833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1415fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.590+0000 7fac137fe640 1 --2- 172.21.15.161:0/2587470833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 0x7fac14072a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fac040097a0 tx=0x7fac0402b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26c2d38de6937c1b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.590+0000 7fac127fc640 1 -- 172.21.15.161:0/2587470833 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0402e070 con 0x7fac14145080 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.590+0000 7fac127fc640 1 -- 172.21.15.161:0/2587470833 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac040315a0 con 0x7fac14145080 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.590+0000 7fac127fc640 1 -- 172.21.15.161:0/2587470833 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04031930 con 0x7fac14145080 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.590+0000 7fac19c07640 1 -- 172.21.15.161:0/2587470833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 msgr2=0x7fac14072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.590+0000 7fac19c07640 1 --2- 172.21.15.161:0/2587470833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 0x7fac14072a00 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fac040097a0 tx=0x7fac0402b840 comp rx=0 tx=0).stop 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.591+0000 7fac19c07640 1 -- 172.21.15.161:0/2587470833 shutdown_connections 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.591+0000 7fac19c07640 1 --2- 172.21.15.161:0/2587470833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1415fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.591+0000 7fac19c07640 1 --2- 172.21.15.161:0/2587470833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 0x7fac14072a00 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.591+0000 7fac19c07640 1 -- 172.21.15.161:0/2587470833 >> 172.21.15.161:0/2587470833 conn(0x7fac140b17f0 msgr2=0x7fac140b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:03.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.591+0000 7fac19c07640 1 -- 172.21.15.161:0/2587470833 shutdown_connections 2024-03-23T14:36:03.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.591+0000 7fac19c07640 1 -- 172.21.15.161:0/2587470833 wait complete. 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.592+0000 7fac19c07640 1 Processor -- start 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.592+0000 7fac19c07640 1 -- start start 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac19c07640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac1406cbf0 0x7fac14159120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac19c07640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 0x7fac14140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac19c07640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1413c040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac19c07640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fac141ef5e0 con 0x7fac1406cbf0 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac19c07640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fac141f2cf0 con 0x7fac14145080 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac19c07640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fac141f20a0 con 0x7fac141607d0 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac1406cbf0 0x7fac14159120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac1406cbf0 0x7fac14159120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38162/0 (socket says 172.21.15.161:38162) 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 -- 172.21.15.161:0/370226286 learned_addr learned my addr 172.21.15.161:0/370226286 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac12ffd640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 0x7fac14140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac13fff640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1413c040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 -- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 msgr2=0x7fac1413c040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1413c040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 -- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 msgr2=0x7fac14140070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 0x7fac14140070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 -- 172.21.15.161:0/370226286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac08009400 con 0x7fac1406cbf0 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fac137fe640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac1406cbf0 0x7fac14159120 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fac04034040 tx=0x7fac040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.593+0000 7fabfbfff640 1 -- 172.21.15.161:0/370226286 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04004230 con 0x7fac1406cbf0 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.594+0000 7fabfbfff640 1 -- 172.21.15.161:0/370226286 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fac04004390 con 0x7fac1406cbf0 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.594+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac04009400 con 0x7fac1406cbf0 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.594+0000 7fabfbfff640 1 -- 172.21.15.161:0/370226286 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04003830 con 0x7fac1406cbf0 2024-03-23T14:36:03.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.594+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac141f2660 con 0x7fac1406cbf0 2024-03-23T14:36:03.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.594+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac141f2e60 con 0x7fac1406cbf0 2024-03-23T14:36:03.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.595+0000 7fabfbfff640 1 -- 172.21.15.161:0/370226286 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fac04003990 con 0x7fac1406cbf0 2024-03-23T14:36:03.592 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.595+0000 7fabfbfff640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fac0006b810 0x7fac0006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.595+0000 7fabfbfff640 1 -- 172.21.15.161:0/370226286 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7fac040ac000 con 0x7fac1406cbf0 2024-03-23T14:36:03.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.596+0000 7fac12ffd640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fac0006b810 0x7fac0006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.596+0000 7fac12ffd640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fac0006b810 0x7fac0006dcd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fac1413fdc0 tx=0x7fac08009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.600+0000 7fabfbfff640 1 -- 172.21.15.161:0/370226286 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fac141f2e60 con 0x7fac1406cbf0 2024-03-23T14:36:03.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.771+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fac140735e0 con 0x7fac1406cbf0 2024-03-23T14:36:03.767 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.772+0000 7fabfbfff640 1 -- 172.21.15.161:0/370226286 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) ==== 74+0+12927 (secure 0 0 0) 0x7fac04076f40 con 0x7fac1406cbf0 2024-03-23T14:36:03.767 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:03.768 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":29,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:00.908442+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":18,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:03.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.773+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fac0006b810 msgr2=0x7fac0006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.773+0000 7fac19c07640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fac0006b810 0x7fac0006dcd0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fac1413fdc0 tx=0x7fac08009190 comp rx=0 tx=0).stop 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.773+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac1406cbf0 msgr2=0x7fac14159120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.773+0000 7fac19c07640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac1406cbf0 0x7fac14159120 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fac04034040 tx=0x7fac040046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.773+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 shutdown_connections 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.773+0000 7fac19c07640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fac0006b810 0x7fac0006dcd0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.773+0000 7fac19c07640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fac141607d0 0x7fac1413c040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.774+0000 7fac19c07640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fac14145080 0x7fac14140070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.774+0000 7fac19c07640 1 --2- 172.21.15.161:0/370226286 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fac1406cbf0 0x7fac14159120 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.774+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 >> 172.21.15.161:0/370226286 conn(0x7fac140b17f0 msgr2=0x7fac14072900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.774+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 shutdown_connections 2024-03-23T14:36:03.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.774+0000 7fac19c07640 1 -- 172.21.15.161:0/370226286 wait complete. 2024-03-23T14:36:03.780 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:03.781 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.889+0000 7f1b3bd25640 1 Processor -- start 2024-03-23T14:36:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.889+0000 7f1b3bd25640 1 -- start start 2024-03-23T14:36:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.890+0000 7f1b3bd25640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1b340cdf30 0x7f1b3415cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.890+0000 7f1b3bd25640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b340ce3b0 con 0x7f1b34145570 2024-03-23T14:36:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.890+0000 7f1b3bd25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b340cf200 con 0x7f1b340d0380 2024-03-23T14:36:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.890+0000 7f1b3bd25640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f1b3415d330 con 0x7f1b340cdf30 2024-03-23T14:36:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.890+0000 7f1b39ac1640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1b34145570 0x7f1b340cfc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38956/0 (socket says 172.21.15.161:38956) 2024-03-23T14:36:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.890+0000 7f1b39ac1640 1 -- 172.21.15.161:0/2742536877 learned_addr learned my addr 172.21.15.161:0/2742536877 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.890+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3724121289 0 0) 0x7f1b30001110 con 0x7f1b34145570 2024-03-23T14:36:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b20003520 con 0x7f1b34145570 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b3a2c2640 1 --2- 172.21.15.161:0/2742536877 >> v2:172.21.15.183:3300/0 conn(0x7f1b340cdf30 0x7f1b3415cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1889291506 0 0) 0x7f1b20003520 con 0x7f1b34145570 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 >> v2:172.21.15.183:3300/0 conn(0x7f1b340cdf30 msgr2=0x7f1b3415cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 --2- 172.21.15.161:0/2742536877 >> v2:172.21.15.183:3300/0 conn(0x7f1b340cdf30 0x7f1b3415cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 >> v1:172.21.15.183:6789/0 conn(0x7f1b340d0380 legacy=0x7f1b340b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b340ce5f0 con 0x7f1b34145570 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f1b30001800 con 0x7f1b34145570 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 >> v1:172.21.15.161:6789/0 conn(0x7f1b34145570 legacy=0x7f1b340cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b20003970 0x7f1b20003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b200047c0 0x7f1b20004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b2000b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3415d330 con 0x7f1b20005180 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1b2000c020 con 0x7f1b20003970 2024-03-23T14:36:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1b2000c1e0 con 0x7f1b200047c0 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b3a2c2640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b2000b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b39ac1640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b20003970 0x7f1b20003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b392c0640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b200047c0 0x7f1b20004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b3a2c2640 1 -- 172.21.15.161:0/2742536877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b200047c0 msgr2=0x7f1b20004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b3a2c2640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b200047c0 0x7f1b20004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b3a2c2640 1 -- 172.21.15.161:0/2742536877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b20003970 msgr2=0x7f1b20003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b3a2c2640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b20003970 0x7f1b20003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.891+0000 7f1b3a2c2640 1 -- 172.21.15.161:0/2742536877 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b20003520 con 0x7f1b20005180 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.892+0000 7f1b3a2c2640 1 -- 172.21.15.161:0/2742536877 reap_dead start 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.892+0000 7f1b3a2c2640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b2000b980 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009fc0 tx=0x7f1b200057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f1c79bd70888106 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.892+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c02cc60 con 0x7f1b20005180 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.892+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b2c00f040 con 0x7f1b20005180 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.892+0000 7f1b38abf640 1 -- 172.21.15.161:0/2742536877 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b2c003990 con 0x7f1b20005180 2024-03-23T14:36:03.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.893+0000 7f1b3bd25640 1 -- 172.21.15.161:0/2742536877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 msgr2=0x7f1b2000b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.893+0000 7f1b3bd25640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b2000b980 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1b2c009fc0 tx=0x7f1b200057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.893+0000 7f1b3bd25640 1 -- 172.21.15.161:0/2742536877 shutdown_connections 2024-03-23T14:36:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.893+0000 7f1b3bd25640 1 --2- 172.21.15.161:0/2742536877 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b2000b980 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.893+0000 7f1b3bd25640 1 -- 172.21.15.161:0/2742536877 >> 172.21.15.161:0/2742536877 conn(0x7f1b3406b260 msgr2=0x7f1b3406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.893+0000 7f1b3bd25640 1 -- 172.21.15.161:0/2742536877 shutdown_connections 2024-03-23T14:36:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.893+0000 7f1b3bd25640 1 -- 172.21.15.161:0/2742536877 wait complete. 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.894+0000 7f1b3bd25640 1 Processor -- start 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.894+0000 7f1b3bd25640 1 -- start start 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3bd25640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b340ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3bd25640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b340addf0 0x7f1b3415ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3bd25640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b3415b250 0x7f1b341410b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3bd25640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1b341f7fa0 con 0x7f1b20005180 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3bd25640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1b3413f970 con 0x7f1b340addf0 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3bd25640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1b341f7cc0 con 0x7f1b3415b250 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b340ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b340ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38188/0 (socket says 172.21.15.161:38188) 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 -- 172.21.15.161:0/566258609 learned_addr learned my addr 172.21.15.161:0/566258609 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:03.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3a2c2640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b3415b250 0x7f1b341410b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b392c0640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b340addf0 0x7f1b3415ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 -- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b3415b250 msgr2=0x7f1b341410b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b3415b250 0x7f1b341410b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 -- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b340addf0 msgr2=0x7f1b3415ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b340addf0 0x7f1b3415ad10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 -- 172.21.15.161:0/566258609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b3413fc10 con 0x7f1b20005180 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3a2c2640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b3415b250 0x7f1b341410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b39ac1640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b340ad8b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1b30007060 tx=0x7f1b30007520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b2a7fc640 1 -- 172.21.15.161:0/566258609 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b30007770 con 0x7f1b20005180 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b2a7fc640 1 -- 172.21.15.161:0/566258609 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3000f9c0 con 0x7f1b20005180 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b3413fef0 con 0x7f1b20005180 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b2a7fc640 1 -- 172.21.15.161:0/566258609 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3000fd50 con 0x7f1b20005180 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.895+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b341417d0 con 0x7f1b20005180 2024-03-23T14:36:03.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.896+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b3413f7d0 con 0x7f1b20005180 2024-03-23T14:36:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.897+0000 7f1b2a7fc640 1 -- 172.21.15.161:0/566258609 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1b3001d460 con 0x7f1b20005180 2024-03-23T14:36:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.897+0000 7f1b2a7fc640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1b1806b840 0x7f1b1806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.897+0000 7f1b2a7fc640 1 -- 172.21.15.161:0/566258609 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f1b30089e40 con 0x7f1b20005180 2024-03-23T14:36:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.897+0000 7f1b392c0640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1b1806b840 0x7f1b1806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.898+0000 7f1b392c0640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1b1806b840 0x7f1b1806dd00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1b3406dc00 tx=0x7f1b24009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:03.901+0000 7f1b2a7fc640 1 -- 172.21.15.161:0/566258609 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1b3413f7d0 con 0x7f1b20005180 2024-03-23T14:36:04.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.072+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1b340ce3b0 con 0x7f1b20005180 2024-03-23T14:36:04.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.072+0000 7f1b2a7fc640 1 -- 172.21.15.161:0/566258609 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v29) ==== 93+0+10 (secure 0 0 0) 0x7f1b340ce3b0 con 0x7f1b20005180 2024-03-23T14:36:04.068 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:04.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1b1806b840 msgr2=0x7f1b1806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1b1806b840 0x7f1b1806dd00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1b3406dc00 tx=0x7f1b24009040 comp rx=0 tx=0).stop 2024-03-23T14:36:04.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 msgr2=0x7f1b340ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b340ad8b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f1b30007060 tx=0x7f1b30007520 comp rx=0 tx=0).stop 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 shutdown_connections 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1b1806b840 0x7f1b1806dd00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1b3415b250 0x7f1b341410b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1b340addf0 0x7f1b3415ad10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 --2- 172.21.15.161:0/566258609 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1b20005180 0x7f1b340ad8b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 >> 172.21.15.161:0/566258609 conn(0x7f1b3406b260 msgr2=0x7f1b341447c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 shutdown_connections 2024-03-23T14:36:04.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.074+0000 7f1b3bd25640 1 -- 172.21.15.161:0/566258609 wait complete. 2024-03-23T14:36:04.081 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T14:36:04.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.185+0000 7f76851a7640 1 Processor -- start 2024-03-23T14:36:04.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.185+0000 7f76851a7640 1 -- start start 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.185+0000 7f76851a7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f768006cd20 0x7f768015e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.185+0000 7f76851a7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7680070ee0 con 0x7f768015a2b0 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.185+0000 7f76851a7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76800b33b0 con 0x7f7680071370 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.186+0000 7f76851a7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f76800b3650 con 0x7f768006cd20 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.186+0000 7f767e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f768006cd20 0x7f768015e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.186+0000 7f767ed9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f768015a2b0 0x7f768015df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38964/0 (socket says 172.21.15.161:38964) 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.186+0000 7f767ed9d640 1 -- 172.21.15.161:0/942741179 learned_addr learned my addr 172.21.15.161:0/942741179 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.186+0000 7f767e59c640 1 -- 172.21.15.161:0/942741179 >> v1:172.21.15.183:6789/0 conn(0x7f7680071370 legacy=0x7f76800717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.186+0000 7f767e59c640 1 -- 172.21.15.161:0/942741179 >> v1:172.21.15.161:6789/0 conn(0x7f768015a2b0 legacy=0x7f768015df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.186+0000 7f767e59c640 1 -- 172.21.15.161:0/942741179 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76800b3930 con 0x7f768006cd20 2024-03-23T14:36:04.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.186+0000 7f767dd9b640 1 -- 172.21.15.161:0/942741179 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 211486452 0 0) 0x7f7680070ee0 con 0x7f768015a2b0 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f767e59c640 1 --2- 172.21.15.161:0/942741179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768006cd20 0x7f768015e690 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f76740097a0 tx=0x7f767402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98cdfa3f929b3d23 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f767dd9b640 1 -- 172.21.15.161:0/942741179 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674031830 con 0x7f768006cd20 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f767dd9b640 1 -- 172.21.15.161:0/942741179 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7674031990 con 0x7f768006cd20 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f767dd9b640 1 -- 172.21.15.161:0/942741179 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7674031d20 con 0x7f768006cd20 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f76851a7640 1 -- 172.21.15.161:0/942741179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768006cd20 msgr2=0x7f768015e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f76851a7640 1 --2- 172.21.15.161:0/942741179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768006cd20 0x7f768015e690 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f76740097a0 tx=0x7f767402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f76851a7640 1 -- 172.21.15.161:0/942741179 shutdown_connections 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f76851a7640 1 --2- 172.21.15.161:0/942741179 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768006cd20 0x7f768015e690 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f76851a7640 1 -- 172.21.15.161:0/942741179 >> 172.21.15.161:0/942741179 conn(0x7f768006b260 msgr2=0x7f768006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f76851a7640 1 -- 172.21.15.161:0/942741179 shutdown_connections 2024-03-23T14:36:04.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.187+0000 7f76851a7640 1 -- 172.21.15.161:0/942741179 wait complete. 2024-03-23T14:36:04.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.189+0000 7f76851a7640 1 Processor -- start 2024-03-23T14:36:04.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.189+0000 7f76851a7640 1 -- start start 2024-03-23T14:36:04.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.189+0000 7f76851a7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f768006cd20 0x7f76800af880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.189+0000 7f76851a7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7680071370 0x7f76800b1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.189+0000 7f76851a7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768015a2b0 0x7f76800b2160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.189+0000 7f76851a7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7680149840 con 0x7f768015a2b0 2024-03-23T14:36:04.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.189+0000 7f76851a7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7680149ae0 con 0x7f7680071370 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.189+0000 7f76851a7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7680149dc0 con 0x7f768006cd20 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768015a2b0 0x7f76800b2160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768015a2b0 0x7f76800b2160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38194/0 (socket says 172.21.15.161:38194) 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 -- 172.21.15.161:0/2001926374 learned_addr learned my addr 172.21.15.161:0/2001926374 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767ed9d640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f768006cd20 0x7f76800af880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767e59c640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7680071370 0x7f76800b1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 -- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f768006cd20 msgr2=0x7f76800af880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f768006cd20 0x7f76800af880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 -- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7680071370 msgr2=0x7f76800b1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7680071370 0x7f76800b1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 -- 172.21.15.161:0/2001926374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7674009400 con 0x7f768015a2b0 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767e59c640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7680071370 0x7f76800b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f767f59e640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768015a2b0 0x7f76800b2160 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f767800c780 tx=0x7f767800cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:04.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f76677fe640 1 -- 172.21.15.161:0/2001926374 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7678003d80 con 0x7f768015a2b0 2024-03-23T14:36:04.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f76677fe640 1 -- 172.21.15.161:0/2001926374 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f767800ce90 con 0x7f768015a2b0 2024-03-23T14:36:04.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f768014a0a0 con 0x7f768015a2b0 2024-03-23T14:36:04.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f76677fe640 1 -- 172.21.15.161:0/2001926374 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76780052b0 con 0x7f768015a2b0 2024-03-23T14:36:04.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.190+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f768014a660 con 0x7f768015a2b0 2024-03-23T14:36:04.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.191+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7680068680 con 0x7f768015a2b0 2024-03-23T14:36:04.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.192+0000 7f76677fe640 1 -- 172.21.15.161:0/2001926374 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7678005410 con 0x7f768015a2b0 2024-03-23T14:36:04.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.192+0000 7f76677fe640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f766c06b810 0x7f766c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.192+0000 7f76677fe640 1 -- 172.21.15.161:0/2001926374 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f7678089690 con 0x7f768015a2b0 2024-03-23T14:36:04.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.192+0000 7f767ed9d640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f766c06b810 0x7f766c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.193+0000 7f767ed9d640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f766c06b810 0x7f766c06dcd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7670007fd0 tx=0x7f7670007de0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:04.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.196+0000 7f76677fe640 1 -- 172.21.15.161:0/2001926374 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f76780544b0 con 0x7f768015a2b0 2024-03-23T14:36:04.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.369+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f768004f4e0 con 0x7f768015a2b0 2024-03-23T14:36:04.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.369+0000 7f76677fe640 1 -- 172.21.15.161:0/2001926374 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) ==== 84+0+65 (secure 0 0 0) 0x7f7678017060 con 0x7f768015a2b0 2024-03-23T14:36:04.365 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:04.365 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-23T14:36:04.365 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:04.365 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-23T14:36:04.366 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-23T14:36:04.366 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:04.366 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-23T14:36:04.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f766c06b810 msgr2=0x7f766c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f766c06b810 0x7f766c06dcd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f7670007fd0 tx=0x7f7670007de0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768015a2b0 msgr2=0x7f76800b2160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768015a2b0 0x7f76800b2160 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f767800c780 tx=0x7f767800cc40 comp rx=0 tx=0).stop 2024-03-23T14:36:04.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 shutdown_connections 2024-03-23T14:36:04.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f766c06b810 0x7f766c06dcd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f768015a2b0 0x7f76800b2160 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7680071370 0x7f76800b1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 --2- 172.21.15.161:0/2001926374 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f768006cd20 0x7f76800af880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 >> 172.21.15.161:0/2001926374 conn(0x7f768006b260 msgr2=0x7f768006b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:04.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 shutdown_connections 2024-03-23T14:36:04.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.371+0000 7f76851a7640 1 -- 172.21.15.161:0/2001926374 wait complete. 2024-03-23T14:36:04.378 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T14:36:04.378 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T14:36:04.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.487+0000 7f8aac09d640 1 Processor -- start 2024-03-23T14:36:04.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.487+0000 7f8aac09d640 1 -- start start 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aac09d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa4070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aac09d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8aa40d0160 0x7f8aa40ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aac09d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8aa4144e80 con 0x7f8aa40710d0 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aac09d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8aa40b3800 con 0x7f8aa40d0160 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aac09d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8aa40b3ae0 con 0x7f8aa4144aa0 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aaa63a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8aa40d0160 0x7f8aa40ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aaa63a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8aa40d0160 0x7f8aa40ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38198/0 (socket says 172.21.15.161:38198) 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aaa63a640 1 -- 172.21.15.161:0/3009124011 learned_addr learned my addr 172.21.15.161:0/3009124011 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:04.485 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aa9e39640 1 --2- 172.21.15.161:0/3009124011 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa4070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aaa63a640 1 -- 172.21.15.161:0/3009124011 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 msgr2=0x7f8aa4070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aaa63a640 1 --2- 172.21.15.161:0/3009124011 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa4070b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aaa63a640 1 -- 172.21.15.161:0/3009124011 >> v1:172.21.15.183:6789/0 conn(0x7f8aa40710d0 legacy=0x7f8aa40cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aaa63a640 1 -- 172.21.15.161:0/3009124011 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8aa40ce4a0 con 0x7f8aa40d0160 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.488+0000 7f8aa9e39640 1 --2- 172.21.15.161:0/3009124011 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa4070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.489+0000 7f8aaa63a640 1 --2- 172.21.15.161:0/3009124011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40d0160 0x7f8aa40ceca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009e50 tx=0x7f8a9c02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27e66b04718c4cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.489+0000 7f8aa8e37640 1 -- 172.21.15.161:0/3009124011 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c02c950 con 0x7f8aa40d0160 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.489+0000 7f8aa8e37640 1 -- 172.21.15.161:0/3009124011 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a9c02cab0 con 0x7f8aa40d0160 2024-03-23T14:36:04.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.489+0000 7f8aa8e37640 1 -- 172.21.15.161:0/3009124011 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c02ce40 con 0x7f8aa40d0160 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.490+0000 7f8aac09d640 1 -- 172.21.15.161:0/3009124011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40d0160 msgr2=0x7f8aa40ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.490+0000 7f8aac09d640 1 --2- 172.21.15.161:0/3009124011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40d0160 0x7f8aa40ceca0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c009e50 tx=0x7f8a9c02bdb0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.490+0000 7f8aac09d640 1 -- 172.21.15.161:0/3009124011 shutdown_connections 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.490+0000 7f8aac09d640 1 --2- 172.21.15.161:0/3009124011 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40d0160 0x7f8aa40ceca0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.490+0000 7f8aac09d640 1 --2- 172.21.15.161:0/3009124011 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa4070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.490+0000 7f8aac09d640 1 -- 172.21.15.161:0/3009124011 >> 172.21.15.161:0/3009124011 conn(0x7f8aa406b080 msgr2=0x7f8aa406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.490+0000 7f8aac09d640 1 -- 172.21.15.161:0/3009124011 shutdown_connections 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.490+0000 7f8aac09d640 1 -- 172.21.15.161:0/3009124011 wait complete. 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.491+0000 7f8aac09d640 1 Processor -- start 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.491+0000 7f8aac09d640 1 -- start start 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aac09d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aa40d0160 0x7f8aa40ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.490 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aac09d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa40c8920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aac09d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40c67f0 0x7f8aa41f2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aac09d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa40c9400 con 0x7f8aa40c67f0 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aac09d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8aa40c96e0 con 0x7f8aa40d0160 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aac09d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8aa40c99c0 con 0x7f8aa4144aa0 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40c67f0 0x7f8aa41f2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40c67f0 0x7f8aa41f2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38210/0 (socket says 172.21.15.161:38210) 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 -- 172.21.15.161:0/596981562 learned_addr learned my addr 172.21.15.161:0/596981562 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aa9638640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa40c8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aa9e39640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aa40d0160 0x7f8aa40ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 -- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 msgr2=0x7f8aa40c8920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa40c8920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 -- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aa40d0160 msgr2=0x7f8aa40ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aa40d0160 0x7f8aa40ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 -- 172.21.15.161:0/596981562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c009b30 con 0x7f8aa40c67f0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aaa63a640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40c67f0 0x7f8aa41f2370 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c036040 tx=0x7f8a9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8a9a7fc640 1 -- 172.21.15.161:0/596981562 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c03e450 con 0x7f8aa40c67f0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8a9a7fc640 1 -- 172.21.15.161:0/596981562 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a9c03e5d0 con 0x7f8aa40c67f0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8aa40c9ca0 con 0x7f8aa40c67f0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8a9a7fc640 1 -- 172.21.15.161:0/596981562 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9c03e980 con 0x7f8aa40c67f0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.492+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8aa40ca9e0 con 0x7f8aa40c67f0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.493+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8aa40ca160 con 0x7f8aa40c67f0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.494+0000 7f8a9a7fc640 1 -- 172.21.15.161:0/596981562 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8a9c03eae0 con 0x7f8aa40c67f0 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.494+0000 7f8a9a7fc640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8a8806b810 0x7f8a8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:04.492 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.494+0000 7f8a9a7fc640 1 -- 172.21.15.161:0/596981562 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(29..29 src has 1..29) ==== 4857+0+0 (secure 0 0 0) 0x7f8a9c0accd0 con 0x7f8aa40c67f0 2024-03-23T14:36:04.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.494+0000 7f8aa9e39640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8a8806b810 0x7f8a8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:04.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.495+0000 7f8aa9e39640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8a8806b810 0x7f8a8806dcd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000aac0 tx=0x7f8aa0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:04.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.498+0000 7f8a9a7fc640 1 -- 172.21.15.161:0/596981562 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8a9c077af0 con 0x7f8aa40c67f0 2024-03-23T14:36:04.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.669+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f8aa40735e0 con 0x7f8aa40c67f0 2024-03-23T14:36:04.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.913+0000 7f8a9a7fc640 1 -- 172.21.15.161:0/596981562 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v30) ==== 138+0+0 (secure 0 0 0) 0x7f8a9c07b1b0 con 0x7f8aa40c67f0 2024-03-23T14:36:04.909 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8a8806b810 msgr2=0x7f8a8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8a8806b810 0x7f8a8806dcd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f8aa000aac0 tx=0x7f8aa0008040 comp rx=0 tx=0).stop 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40c67f0 msgr2=0x7f8aa41f2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40c67f0 0x7f8aa41f2370 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c036040 tx=0x7f8a9c004a40 comp rx=0 tx=0).stop 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 shutdown_connections 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8a8806b810 0x7f8a8806dcd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8aa40c67f0 0x7f8aa41f2370 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8aa4144aa0 0x7f8aa40c8920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.912 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 --2- 172.21.15.161:0/596981562 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8aa40d0160 0x7f8aa40ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:04.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.916+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 >> 172.21.15.161:0/596981562 conn(0x7f8aa406b080 msgr2=0x7f8aa415b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:04.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.917+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 shutdown_connections 2024-03-23T14:36:04.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:04.917+0000 7f8aac09d640 1 -- 172.21.15.161:0/596981562 wait complete. 2024-03-23T14:36:04.927 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T14:36:05.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.038+0000 7fd43c6fb640 1 Processor -- start 2024-03-23T14:36:05.034 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.039+0000 7fd43c6fb640 1 -- start start 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.039+0000 7fd43c6fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 0x7fd4340d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.039+0000 7fd43c6fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4340d2bf0 0x7fd4340b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.039+0000 7fd43c6fb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4340b0ac0 con 0x7fd4340d1d50 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.039+0000 7fd43c6fb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd4340b4220 con 0x7fd4340d2bf0 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.039+0000 7fd43c6fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd4340b3350 con 0x7fd43415e440 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.040+0000 7fd43ac98640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4340d2bf0 0x7fd4340b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.040+0000 7fd43ac98640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd4340d2bf0 0x7fd4340b46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38222/0 (socket says 172.21.15.161:38222) 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.040+0000 7fd43ac98640 1 -- 172.21.15.161:0/3405506049 learned_addr learned my addr 172.21.15.161:0/3405506049 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:05.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.040+0000 7fd43a497640 1 --2- 172.21.15.161:0/3405506049 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 0x7fd4340d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:05.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.040+0000 7fd43ac98640 1 -- 172.21.15.161:0/3405506049 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 msgr2=0x7fd4340d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:05.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.040+0000 7fd43ac98640 1 --2- 172.21.15.161:0/3405506049 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 0x7fd4340d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.040+0000 7fd43ac98640 1 -- 172.21.15.161:0/3405506049 >> v1:172.21.15.183:6789/0 conn(0x7fd4340d1d50 legacy=0x7fd4340d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:05.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.040+0000 7fd43ac98640 1 -- 172.21.15.161:0/3405506049 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4340b3610 con 0x7fd4340d2bf0 2024-03-23T14:36:05.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.041+0000 7fd43ac98640 1 --2- 172.21.15.161:0/3405506049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340b46b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009e50 tx=0x7fd42c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c32357717e05c17 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:05.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.041+0000 7fd439495640 1 -- 172.21.15.161:0/3405506049 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c02cb00 con 0x7fd4340d2bf0 2024-03-23T14:36:05.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.041+0000 7fd439495640 1 -- 172.21.15.161:0/3405506049 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd42c02cc60 con 0x7fd4340d2bf0 2024-03-23T14:36:05.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.041+0000 7fd439495640 1 -- 172.21.15.161:0/3405506049 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd42c0343f0 con 0x7fd4340d2bf0 2024-03-23T14:36:05.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.041+0000 7fd43c6fb640 1 -- 172.21.15.161:0/3405506049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 msgr2=0x7fd4340b46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:05.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.041+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/3405506049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340b46b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fd42c009e50 tx=0x7fd42c02bf60 comp rx=0 tx=0).stop 2024-03-23T14:36:05.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.042+0000 7fd43c6fb640 1 -- 172.21.15.161:0/3405506049 shutdown_connections 2024-03-23T14:36:05.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.042+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/3405506049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340b46b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.042+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/3405506049 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 0x7fd4340d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.042+0000 7fd43c6fb640 1 -- 172.21.15.161:0/3405506049 >> 172.21.15.161:0/3405506049 conn(0x7fd4340d3f00 msgr2=0x7fd4340d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:05.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.042+0000 7fd43c6fb640 1 -- 172.21.15.161:0/3405506049 shutdown_connections 2024-03-23T14:36:05.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.042+0000 7fd43c6fb640 1 -- 172.21.15.161:0/3405506049 wait complete. 2024-03-23T14:36:05.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.043+0000 7fd43c6fb640 1 Processor -- start 2024-03-23T14:36:05.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.043+0000 7fd43c6fb640 1 -- start start 2024-03-23T14:36:05.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.043+0000 7fd43c6fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4340d1d50 0x7fd4341f5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:05.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.043+0000 7fd43c6fb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340cbd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:05.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.043+0000 7fd43c6fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 0x7fd4340b5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:05.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.043+0000 7fd43c6fb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341f8700 con 0x7fd4340d2bf0 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.043+0000 7fd43c6fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd4341f7df0 con 0x7fd4340d1d50 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.043+0000 7fd43c6fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd4340cc710 con 0x7fd43415e440 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340cbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340cbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38224/0 (socket says 172.21.15.161:38224) 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 -- 172.21.15.161:0/1535362882 learned_addr learned my addr 172.21.15.161:0/1535362882 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd43a497640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4340d1d50 0x7fd4341f5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd43ac98640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 0x7fd4340b5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 -- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 msgr2=0x7fd4340b5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 0x7fd4340b5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 -- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4340d1d50 msgr2=0x7fd4341f5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4340d1d50 0x7fd4341f5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 -- 172.21.15.161:0/1535362882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42c009b30 con 0x7fd4340d2bf0 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd439c96640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340cbd30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd42400c440 tx=0x7fd424001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:05.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd42affd640 1 -- 172.21.15.161:0/1535362882 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4240052d0 con 0x7fd4340d2bf0 2024-03-23T14:36:05.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd42affd640 1 -- 172.21.15.161:0/1535362882 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd424005430 con 0x7fd4340d2bf0 2024-03-23T14:36:05.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd4341f80b0 con 0x7fd4340d2bf0 2024-03-23T14:36:05.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd42affd640 1 -- 172.21.15.161:0/1535362882 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd424007260 con 0x7fd4340d2bf0 2024-03-23T14:36:05.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.044+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4340cb360 con 0x7fd4340d2bf0 2024-03-23T14:36:05.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.045+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd434068160 con 0x7fd4340d2bf0 2024-03-23T14:36:05.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.046+0000 7fd42affd640 1 -- 172.21.15.161:0/1535362882 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd4240073c0 con 0x7fd4340d2bf0 2024-03-23T14:36:05.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.046+0000 7fd42affd640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41806b840 0x7fd41806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:05.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.046+0000 7fd42affd640 1 -- 172.21.15.161:0/1535362882 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(30..30 src has 1..30) ==== 5227+0+0 (secure 0 0 0) 0x7fd424089350 con 0x7fd4340d2bf0 2024-03-23T14:36:05.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.046+0000 7fd43a497640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41806b840 0x7fd41806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:05.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.047+0000 7fd43a497640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41806b840 0x7fd41806dd00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4340b55d0 tx=0x7fd430009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:05.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.051+0000 7fd42affd640 1 -- 172.21.15.161:0/1535362882 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd424054000 con 0x7fd4340d2bf0 2024-03-23T14:36:05.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.222+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fd4340d22c0 con 0x7fd4340d2bf0 2024-03-23T14:36:05.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.916+0000 7fd42affd640 1 -- 172.21.15.161:0/1535362882 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v31) ==== 170+0+0 (secure 0 0 0) 0x7fd4240576c0 con 0x7fd4340d2bf0 2024-03-23T14:36:05.912 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-23T14:36:05.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.920+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41806b840 msgr2=0x7fd41806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:05.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.920+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41806b840 0x7fd41806dd00 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd4340b55d0 tx=0x7fd430009190 comp rx=0 tx=0).stop 2024-03-23T14:36:05.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.920+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 msgr2=0x7fd4340cbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:05.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.920+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340cbd30 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fd42400c440 tx=0x7fd424001e70 comp rx=0 tx=0).stop 2024-03-23T14:36:05.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.921+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 shutdown_connections 2024-03-23T14:36:05.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.921+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd41806b840 0x7fd41806dd00 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.921+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd43415e440 0x7fd4340b5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.921+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd4340d2bf0 0x7fd4340cbd30 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.921+0000 7fd43c6fb640 1 --2- 172.21.15.161:0/1535362882 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4340d1d50 0x7fd4341f5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:05.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.921+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 >> 172.21.15.161:0/1535362882 conn(0x7fd4340d3f00 msgr2=0x7fd43406d350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:05.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.923+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 shutdown_connections 2024-03-23T14:36:05.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:05.923+0000 7fd43c6fb640 1 -- 172.21.15.161:0/1535362882 wait complete. 2024-03-23T14:36:05.938 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T14:36:06.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.047+0000 7fa21633e640 1 Processor -- start 2024-03-23T14:36:06.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.047+0000 7fa21633e640 1 -- start start 2024-03-23T14:36:06.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.047+0000 7fa21633e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa210145520 0x7fa210160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:06.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.047+0000 7fa21633e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa2100cfee0 0x7fa2100d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa21633e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2100d0880 con 0x7fa210070c20 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa21633e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa2101608c0 con 0x7fa2100cfee0 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa21633e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa2100ce410 con 0x7fa210145520 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa20f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa2100cfee0 0x7fa2100d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa20f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa2100cfee0 0x7fa2100d0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38240/0 (socket says 172.21.15.161:38240) 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa20f7fe640 1 -- 172.21.15.161:0/766858364 learned_addr learned my addr 172.21.15.161:0/766858364 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa20f7fe640 1 -- 172.21.15.161:0/766858364 >> v2:172.21.15.183:3300/0 conn(0x7fa210145520 msgr2=0x7fa210160380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa20f7fe640 1 --2- 172.21.15.161:0/766858364 >> v2:172.21.15.183:3300/0 conn(0x7fa210145520 0x7fa210160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa20f7fe640 1 -- 172.21.15.161:0/766858364 >> v1:172.21.15.183:6789/0 conn(0x7fa210070c20 legacy=0x7fa210071050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.048+0000 7fa20f7fe640 1 -- 172.21.15.161:0/766858364 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2100ce6f0 con 0x7fa2100cfee0 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa20f7fe640 1 --2- 172.21.15.161:0/766858364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2100cfee0 0x7fa2100d0340 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa21006eea0 tx=0x7fa20402b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82e184e3bc33260e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa20effd640 1 -- 172.21.15.161:0/766858364 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2040317e0 con 0x7fa2100cfee0 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa20effd640 1 -- 172.21.15.161:0/766858364 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa204031940 con 0x7fa2100cfee0 2024-03-23T14:36:06.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa20effd640 1 -- 172.21.15.161:0/766858364 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa204031cd0 con 0x7fa2100cfee0 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa21633e640 1 -- 172.21.15.161:0/766858364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2100cfee0 msgr2=0x7fa2100d0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa21633e640 1 --2- 172.21.15.161:0/766858364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2100cfee0 0x7fa2100d0340 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa21006eea0 tx=0x7fa20402b880 comp rx=0 tx=0).stop 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa21633e640 1 -- 172.21.15.161:0/766858364 shutdown_connections 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa21633e640 1 --2- 172.21.15.161:0/766858364 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa2100cfee0 0x7fa2100d0340 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa21633e640 1 --2- 172.21.15.161:0/766858364 >> v2:172.21.15.183:3300/0 conn(0x7fa210145520 0x7fa210160380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa21633e640 1 -- 172.21.15.161:0/766858364 >> 172.21.15.161:0/766858364 conn(0x7fa21006b4a0 msgr2=0x7fa21006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa21633e640 1 -- 172.21.15.161:0/766858364 shutdown_connections 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.049+0000 7fa21633e640 1 -- 172.21.15.161:0/766858364 wait complete. 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.051+0000 7fa21633e640 1 Processor -- start 2024-03-23T14:36:06.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.051+0000 7fa21633e640 1 -- start start 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.051+0000 7fa21633e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa210070c20 0x7fa2101f15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.051+0000 7fa21633e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2100cfee0 0x7fa2100b16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.051+0000 7fa21633e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa210145520 0x7fa2100b1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.051+0000 7fa21633e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa21012e900 con 0x7fa210070c20 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.051+0000 7fa21633e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa210155b20 con 0x7fa2100cfee0 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.051+0000 7fa21633e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa21012dc80 con 0x7fa210145520 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa210070c20 0x7fa2101f15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa210070c20 0x7fa2101f15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38244/0 (socket says 172.21.15.161:38244) 2024-03-23T14:36:06.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 -- 172.21.15.161:0/319377201 learned_addr learned my addr 172.21.15.161:0/319377201 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20f7fe640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2100cfee0 0x7fa2100b16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 -- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa210145520 msgr2=0x7fa2100b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa2148db640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa210145520 0x7fa2100b1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa210145520 0x7fa2100b1c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 -- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2100cfee0 msgr2=0x7fa2100b16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2100cfee0 0x7fa2100b16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 -- 172.21.15.161:0/319377201 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2040094d0 con 0x7fa210070c20 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20f7fe640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2100cfee0 0x7fa2100b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa2148db640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa210145520 0x7fa2100b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20ffff640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa210070c20 0x7fa2101f15f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa20000b410 tx=0x7fa20000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20cff9640 1 -- 172.21.15.161:0/319377201 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200011830 con 0x7fa210070c20 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20cff9640 1 -- 172.21.15.161:0/319377201 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa2000119b0 con 0x7fa210070c20 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa21012df60 con 0x7fa210070c20 2024-03-23T14:36:06.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa20cff9640 1 -- 172.21.15.161:0/319377201 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa200011d60 con 0x7fa210070c20 2024-03-23T14:36:06.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.052+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa21012e520 con 0x7fa210070c20 2024-03-23T14:36:06.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.053+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa210073570 con 0x7fa210070c20 2024-03-23T14:36:06.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.054+0000 7fa20cff9640 1 -- 172.21.15.161:0/319377201 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa20001a400 con 0x7fa210070c20 2024-03-23T14:36:06.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.054+0000 7fa20cff9640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1fc06b810 0x7fa1fc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:06.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.054+0000 7fa20cff9640 1 -- 172.21.15.161:0/319377201 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(31..31 src has 1..31) ==== 5609+0+0 (secure 0 0 0) 0x7fa20008a2a0 con 0x7fa210070c20 2024-03-23T14:36:06.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.054+0000 7fa20f7fe640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1fc06b810 0x7fa1fc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:06.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.055+0000 7fa20f7fe640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1fc06b810 0x7fa1fc06dcd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa204036000 tx=0x7fa20403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:06.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.059+0000 7fa20cff9640 1 -- 172.21.15.161:0/319377201 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa200054d30 con 0x7fa210070c20 2024-03-23T14:36:06.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.266+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fa21004f550 con 0x7fa210070c20 2024-03-23T14:36:06.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.937+0000 7fa20cff9640 1 -- 172.21.15.161:0/319377201 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 4 and data pool 5 v11) ==== 172+0+0 (secure 0 0 0) 0x7fa2000583f0 con 0x7fa210070c20 2024-03-23T14:36:06.933 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 4 and data pool 5 2024-03-23T14:36:06.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1fc06b810 msgr2=0x7fa1fc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:06.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1fc06b810 0x7fa1fc06dcd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa204036000 tx=0x7fa20403d040 comp rx=0 tx=0).stop 2024-03-23T14:36:06.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa210070c20 msgr2=0x7fa2101f15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa210070c20 0x7fa2101f15f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fa20000b410 tx=0x7fa20000b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 shutdown_connections 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1fc06b810 0x7fa1fc06dcd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa210145520 0x7fa2100b1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa2100cfee0 0x7fa2100b16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 --2- 172.21.15.161:0/319377201 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa210070c20 0x7fa2101f15f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 >> 172.21.15.161:0/319377201 conn(0x7fa21006b4a0 msgr2=0x7fa21013f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.944+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 shutdown_connections 2024-03-23T14:36:06.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:06.945+0000 7fa21633e640 1 -- 172.21.15.161:0/319377201 wait complete. 2024-03-23T14:36:06.959 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:07.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.065+0000 7fca2c69d640 1 Processor -- start 2024-03-23T14:36:07.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.065+0000 7fca2c69d640 1 -- start start 2024-03-23T14:36:07.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.066+0000 7fca2c69d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca2415e400 0x7fca240cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.066+0000 7fca2c69d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fca240cf9d0 0x7fca240d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.066+0000 7fca2c69d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca240d0460 con 0x7fca240d1470 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.066+0000 7fca2c69d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fca240d1050 con 0x7fca240cf9d0 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.066+0000 7fca2c69d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fca240d06b0 con 0x7fca2415e400 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca29c38640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fca240cf9d0 0x7fca240d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca2ac3a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fca240d1470 0x7fca240b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38968/0 (socket says 172.21.15.161:38968) 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca2ac3a640 1 -- 172.21.15.161:0/1947526140 learned_addr learned my addr 172.21.15.161:0/1947526140 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca2a439640 1 --2- 172.21.15.161:0/1947526140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca2415e400 0x7fca240cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca29c38640 1 -- 172.21.15.161:0/1947526140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca2415e400 msgr2=0x7fca240cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:07.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca29c38640 1 --2- 172.21.15.161:0/1947526140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca2415e400 0x7fca240cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca29c38640 1 -- 172.21.15.161:0/1947526140 >> v1:172.21.15.161:6789/0 conn(0x7fca240d1470 legacy=0x7fca240b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca29437640 1 -- 172.21.15.161:0/1947526140 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2311471543 0 0) 0x7fca240d0460 con 0x7fca240d1470 2024-03-23T14:36:07.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca29c38640 1 -- 172.21.15.161:0/1947526140 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca240d0990 con 0x7fca240cf9d0 2024-03-23T14:36:07.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.067+0000 7fca2a439640 1 --2- 172.21.15.161:0/1947526140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca2415e400 0x7fca240cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:07.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca29c38640 1 --2- 172.21.15.161:0/1947526140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca240cf9d0 0x7fca240d2430 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fca140097a0 tx=0x7fca1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95712b651bed9f07 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca29437640 1 -- 172.21.15.161:0/1947526140 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14031830 con 0x7fca240cf9d0 2024-03-23T14:36:07.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca29437640 1 -- 172.21.15.161:0/1947526140 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca14031990 con 0x7fca240cf9d0 2024-03-23T14:36:07.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca29437640 1 -- 172.21.15.161:0/1947526140 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14031d20 con 0x7fca240cf9d0 2024-03-23T14:36:07.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca2c69d640 1 -- 172.21.15.161:0/1947526140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca240cf9d0 msgr2=0x7fca240d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca2c69d640 1 --2- 172.21.15.161:0/1947526140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca240cf9d0 0x7fca240d2430 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fca140097a0 tx=0x7fca1402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca2c69d640 1 -- 172.21.15.161:0/1947526140 shutdown_connections 2024-03-23T14:36:07.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca2c69d640 1 --2- 172.21.15.161:0/1947526140 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca240cf9d0 0x7fca240d2430 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca2c69d640 1 --2- 172.21.15.161:0/1947526140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca2415e400 0x7fca240cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca2c69d640 1 -- 172.21.15.161:0/1947526140 >> 172.21.15.161:0/1947526140 conn(0x7fca2409d190 msgr2=0x7fca240d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:07.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca2c69d640 1 -- 172.21.15.161:0/1947526140 shutdown_connections 2024-03-23T14:36:07.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.068+0000 7fca2c69d640 1 -- 172.21.15.161:0/1947526140 wait complete. 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2c69d640 1 Processor -- start 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2c69d640 1 -- start start 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2c69d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca240cf9d0 0x7fca240cc420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2c69d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca240d1470 0x7fca241f33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2c69d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2415e400 0x7fca24160240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2c69d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fca241f38f0 con 0x7fca2415e400 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2c69d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fca24159840 con 0x7fca240cf9d0 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2c69d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fca240cca90 con 0x7fca240d1470 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2ac3a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2415e400 0x7fca24160240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2ac3a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2415e400 0x7fca24160240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38258/0 (socket says 172.21.15.161:38258) 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2ac3a640 1 -- 172.21.15.161:0/496355250 learned_addr learned my addr 172.21.15.161:0/496355250 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.070+0000 7fca2a439640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca240cf9d0 0x7fca240cc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca29c38640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca240d1470 0x7fca241f33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca2ac3a640 1 -- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca240d1470 msgr2=0x7fca241f33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca2ac3a640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca240d1470 0x7fca241f33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca2ac3a640 1 -- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca240cf9d0 msgr2=0x7fca240cc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca2ac3a640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca240cf9d0 0x7fca240cc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca2ac3a640 1 -- 172.21.15.161:0/496355250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca14009400 con 0x7fca2415e400 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca29c38640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca240d1470 0x7fca241f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca2ac3a640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2415e400 0x7fca24160240 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fca2000c450 tx=0x7fca2000c910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca1affd640 1 -- 172.21.15.161:0/496355250 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca200074e0 con 0x7fca2415e400 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca1affd640 1 -- 172.21.15.161:0/496355250 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca20007660 con 0x7fca2415e400 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca240ccd70 con 0x7fca2415e400 2024-03-23T14:36:07.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca1affd640 1 -- 172.21.15.161:0/496355250 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca20004a20 con 0x7fca2415e400 2024-03-23T14:36:07.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.071+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca240cd330 con 0x7fca2415e400 2024-03-23T14:36:07.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.072+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca240a1e00 con 0x7fca2415e400 2024-03-23T14:36:07.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.072+0000 7fca1affd640 1 -- 172.21.15.161:0/496355250 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fca20004b80 con 0x7fca2415e400 2024-03-23T14:36:07.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.073+0000 7fca1affd640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca0806b810 0x7fca0806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.073+0000 7fca1affd640 1 -- 172.21.15.161:0/496355250 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fca20089540 con 0x7fca2415e400 2024-03-23T14:36:07.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.073+0000 7fca2a439640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca0806b810 0x7fca0806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.073+0000 7fca2a439640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca0806b810 0x7fca0806dcd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00f500 tx=0x7fca1c007770 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.077+0000 7fca1affd640 1 -- 172.21.15.161:0/496355250 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fca200540a0 con 0x7fca2415e400 2024-03-23T14:36:07.241 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:36:07.244 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:36:07.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.250+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fca2404f550 con 0x7fca2415e400 2024-03-23T14:36:07.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.251+0000 7fca1affd640 1 -- 172.21.15.161:0/496355250 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16718 (secure 0 0 0) 0x7fca20057760 con 0x7fca2415e400 2024-03-23T14:36:07.248 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:07.248 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":32,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:06.267431+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca0806b810 msgr2=0x7fca0806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca0806b810 0x7fca0806dcd0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fca1c00f500 tx=0x7fca1c007770 comp rx=0 tx=0).stop 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2415e400 msgr2=0x7fca24160240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2415e400 0x7fca24160240 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fca2000c450 tx=0x7fca2000c910 comp rx=0 tx=0).stop 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 shutdown_connections 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca0806b810 0x7fca0806dcd0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca2415e400 0x7fca24160240 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca240d1470 0x7fca241f33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 --2- 172.21.15.161:0/496355250 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca240cf9d0 0x7fca240cc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 >> 172.21.15.161:0/496355250 conn(0x7fca2409d190 msgr2=0x7fca2415e7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:07.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.253+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 shutdown_connections 2024-03-23T14:36:07.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.254+0000 7fca2c69d640 1 -- 172.21.15.161:0/496355250 wait complete. 2024-03-23T14:36:07.261 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:07.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.447+0000 7f8531f6c640 1 Processor -- start 2024-03-23T14:36:07.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.447+0000 7f8531f6c640 1 -- start start 2024-03-23T14:36:07.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.447+0000 7f8531f6c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f852c06eb30 0x7f852c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.447+0000 7f8531f6c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f852c070b90 con 0x7f852c0b4260 2024-03-23T14:36:07.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.447+0000 7f8531f6c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f852c15b0d0 con 0x7f852c0b3610 2024-03-23T14:36:07.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.447+0000 7f8531f6c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f852c15ab90 con 0x7f852c06eb30 2024-03-23T14:36:07.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f852c0b4260 0x7f852c15b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38972/0 (socket says 172.21.15.161:38972) 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852affd640 1 -- 172.21.15.161:0/4211018612 learned_addr learned my addr 172.21.15.161:0/4211018612 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1206536830 0 0) 0x7f852c070b90 con 0x7f852c0b4260 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852b7fe640 1 --2- 172.21.15.161:0/4211018612 >> v2:172.21.15.183:3300/0 conn(0x7f852c06eb30 0x7f852c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8514003520 con 0x7f852c0b4260 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 48406042 0 0) 0x7f8514003520 con 0x7f852c0b4260 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 >> v2:172.21.15.183:3300/0 conn(0x7f852c06eb30 msgr2=0x7f852c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 --2- 172.21.15.161:0/4211018612 >> v2:172.21.15.183:3300/0 conn(0x7f852c06eb30 0x7f852c0b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 >> v1:172.21.15.183:6789/0 conn(0x7f852c0b3610 legacy=0x7f852c06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852c15ada0 con 0x7f852c0b4260 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f8520001670 con 0x7f852c0b4260 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 >> v1:172.21.15.161:6789/0 conn(0x7f852c0b4260 legacy=0x7f852c15b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514003970 0x7f8514003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85140047c0 0x7f8514004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8514005180 0x7f851400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f852c15ab90 con 0x7f8514005180 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f851400c120 con 0x7f85140047c0 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f851400c2e0 con 0x7f8514003970 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852bfff640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8514005180 0x7f851400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852affd640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85140047c0 0x7f8514004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.448+0000 7f852b7fe640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514003970 0x7f8514003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852bfff640 1 -- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514003970 msgr2=0x7f8514003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852bfff640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514003970 0x7f8514003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852bfff640 1 -- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85140047c0 msgr2=0x7f8514004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852bfff640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f85140047c0 0x7f8514004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852bfff640 1 -- 172.21.15.161:0/4211018612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8514003520 con 0x7f8514005180 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852b7fe640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514003970 0x7f8514003d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852bfff640 1 -- 172.21.15.161:0/4211018612 reap_dead start 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852bfff640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8514005180 0x7f851400b980 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8524001e50 tx=0x7f85140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f7c570f5a7eccfd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524003b70 con 0x7f8514005180 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8524003d10 con 0x7f8514005180 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.449+0000 7f852a7fc640 1 -- 172.21.15.161:0/4211018612 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524005310 con 0x7f8514005180 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.450+0000 7f8531f6c640 1 -- 172.21.15.161:0/4211018612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8514005180 msgr2=0x7f851400b980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.450+0000 7f8531f6c640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8514005180 0x7f851400b980 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8524001e50 tx=0x7f85140057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.450+0000 7f8531f6c640 1 -- 172.21.15.161:0/4211018612 shutdown_connections 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.450+0000 7f8531f6c640 1 --2- 172.21.15.161:0/4211018612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8514005180 0x7f851400b980 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.450+0000 7f8531f6c640 1 -- 172.21.15.161:0/4211018612 >> 172.21.15.161:0/4211018612 conn(0x7f852c0b17f0 msgr2=0x7f852c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.450+0000 7f8531f6c640 1 -- 172.21.15.161:0/4211018612 shutdown_connections 2024-03-23T14:36:07.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.450+0000 7f8531f6c640 1 -- 172.21.15.161:0/4211018612 wait complete. 2024-03-23T14:36:07.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.451+0000 7f8531f6c640 1 Processor -- start 2024-03-23T14:36:07.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.451+0000 7f8531f6c640 1 -- start start 2024-03-23T14:36:07.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f8531f6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514005180 0x7f852c14d8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f8531f6c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c150f20 0x7f852c06ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f8531f6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c13f110 0x7f852c13f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f8531f6c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f852c0ccaa0 con 0x7f852c13f110 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f8531f6c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f852c151d10 con 0x7f852c150f20 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f8531f6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f852c13fad0 con 0x7f8514005180 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c13f110 0x7f852c13f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c13f110 0x7f852c13f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38280/0 (socket says 172.21.15.161:38280) 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852bfff640 1 -- 172.21.15.161:0/3749565922 learned_addr learned my addr 172.21.15.161:0/3749565922 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852affd640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c150f20 0x7f852c06ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852b7fe640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514005180 0x7f852c14d8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852bfff640 1 -- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514005180 msgr2=0x7f852c14d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852bfff640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514005180 0x7f852c14d8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852bfff640 1 -- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c150f20 msgr2=0x7f852c06ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852bfff640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c150f20 0x7f852c06ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852bfff640 1 -- 172.21.15.161:0/3749565922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f852c151470 con 0x7f852c13f110 2024-03-23T14:36:07.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852affd640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c150f20 0x7f852c06ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.452+0000 7f852b7fe640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514005180 0x7f852c14d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.453+0000 7f852bfff640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c13f110 0x7f852c13f590 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8524001f70 tx=0x7f85240014a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.453+0000 7f8513fff640 1 -- 172.21.15.161:0/3749565922 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8524006d50 con 0x7f852c13f110 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.453+0000 7f8513fff640 1 -- 172.21.15.161:0/3749565922 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8524007030 con 0x7f852c13f110 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.453+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f852c151750 con 0x7f852c13f110 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.453+0000 7f8513fff640 1 -- 172.21.15.161:0/3749565922 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85240073e0 con 0x7f852c13f110 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.453+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f852c14de20 con 0x7f852c13f110 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.454+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f852c14dfe0 con 0x7f852c13f110 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.454+0000 7f8513fff640 1 -- 172.21.15.161:0/3749565922 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8524007540 con 0x7f852c13f110 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.454+0000 7f8513fff640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.455+0000 7f8513fff640 1 -- 172.21.15.161:0/3749565922 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7f85240ac270 con 0x7f852c13f110 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.455+0000 7f852b7fe640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.455+0000 7f852b7fe640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f851c00a910 tx=0x7f851c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.459+0000 7f8513fff640 1 -- 172.21.15.161:0/3749565922 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f852c14dfe0 con 0x7f852c13f110 2024-03-23T14:36:07.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.631+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f852c0cc800 con 0x7f852c13f110 2024-03-23T14:36:07.628 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.633+0000 7f8513fff640 1 -- 172.21.15.161:0/3749565922 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) ==== 74+0+16718 (secure 0 0 0) 0x7f852c0cc800 con 0x7f852c13f110 2024-03-23T14:36:07.629 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:07.629 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":32,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:06.267431+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 msgr2=0x7f851806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f851c00a910 tx=0x7f851c008040 comp rx=0 tx=0).stop 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c13f110 msgr2=0x7f852c13f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c13f110 0x7f852c13f590 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8524001f70 tx=0x7f85240014a0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 shutdown_connections 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f851806b810 0x7f851806dcd0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f852c13f110 0x7f852c13f590 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f852c150f20 0x7f852c06ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 --2- 172.21.15.161:0/3749565922 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8514005180 0x7f852c14d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 >> 172.21.15.161:0/3749565922 conn(0x7f852c0b17f0 msgr2=0x7f852c154470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:07.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 shutdown_connections 2024-03-23T14:36:07.632 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.635+0000 7f8531f6c640 1 -- 172.21.15.161:0/3749565922 wait complete. 2024-03-23T14:36:07.642 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T14:36:07.745 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.750+0000 7fdd4defb640 1 Processor -- start 2024-03-23T14:36:07.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.750+0000 7fdd4defb640 1 -- start start 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.750+0000 7fdd4defb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd48145520 0x7fdd48160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.750+0000 7fdd4defb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd48071a40 con 0x7fdd480b3a20 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.750+0000 7fdd4defb640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd481608c0 con 0x7fdd480cfee0 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.750+0000 7fdd4defb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdd480ce410 con 0x7fdd48145520 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.751+0000 7fdd477fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd48145520 0x7fdd48160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.751+0000 7fdd477fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdd48145520 0x7fdd48160380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38292/0 (socket says 172.21.15.161:38292) 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.751+0000 7fdd47fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fdd480b3a20 0x7fdd48071330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:38974/0 (socket says 172.21.15.161:38974) 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.751+0000 7fdd477fe640 1 -- 172.21.15.161:0/3750677534 learned_addr learned my addr 172.21.15.161:0/3750677534 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:07.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.751+0000 7fdd477fe640 1 -- 172.21.15.161:0/3750677534 >> v1:172.21.15.183:6789/0 conn(0x7fdd480cfee0 legacy=0x7fdd48070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.751+0000 7fdd477fe640 1 -- 172.21.15.161:0/3750677534 >> v1:172.21.15.161:6789/0 conn(0x7fdd480b3a20 legacy=0x7fdd48071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.751+0000 7fdd477fe640 1 -- 172.21.15.161:0/3750677534 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd480ce6f0 con 0x7fdd48145520 2024-03-23T14:36:07.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.751+0000 7fdd467fc640 1 -- 172.21.15.161:0/3750677534 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247447158 0 0) 0x7fdd48071a40 con 0x7fdd480b3a20 2024-03-23T14:36:07.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd477fe640 1 --2- 172.21.15.161:0/3750677534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd48160380 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009870 tx=0x7fdd3802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f30114064b92e8c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd467fc640 1 -- 172.21.15.161:0/3750677534 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38030860 con 0x7fdd48145520 2024-03-23T14:36:07.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd467fc640 1 -- 172.21.15.161:0/3750677534 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd380309c0 con 0x7fdd48145520 2024-03-23T14:36:07.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd467fc640 1 -- 172.21.15.161:0/3750677534 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd38030d50 con 0x7fdd48145520 2024-03-23T14:36:07.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd4defb640 1 -- 172.21.15.161:0/3750677534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 msgr2=0x7fdd48160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd4defb640 1 --2- 172.21.15.161:0/3750677534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd48160380 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fdd38009870 tx=0x7fdd3802b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd4defb640 1 -- 172.21.15.161:0/3750677534 shutdown_connections 2024-03-23T14:36:07.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd4defb640 1 --2- 172.21.15.161:0/3750677534 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd48160380 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd4defb640 1 -- 172.21.15.161:0/3750677534 >> 172.21.15.161:0/3750677534 conn(0x7fdd4806b4a0 msgr2=0x7fdd4806b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:07.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd4defb640 1 -- 172.21.15.161:0/3750677534 shutdown_connections 2024-03-23T14:36:07.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.752+0000 7fdd4defb640 1 -- 172.21.15.161:0/3750677534 wait complete. 2024-03-23T14:36:07.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.753+0000 7fdd4defb640 1 Processor -- start 2024-03-23T14:36:07.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd4defb640 1 -- start start 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd4defb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd480b3a20 0x7fdd48157520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd4defb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd480cfee0 0x7fdd480aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd4defb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd4815d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd4defb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4806e460 con 0x7fdd48145520 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd4defb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdd4806e6b0 con 0x7fdd480cfee0 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd4defb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fdd4806e990 con 0x7fdd480b3a20 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd47fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd4815d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.754+0000 7fdd47fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd4815d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38302/0 (socket says 172.21.15.161:38302) 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd47fff640 1 -- 172.21.15.161:0/988870761 learned_addr learned my addr 172.21.15.161:0/988870761 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd46ffd640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd480cfee0 0x7fdd480aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd477fe640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd480b3a20 0x7fdd48157520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd47fff640 1 -- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd480b3a20 msgr2=0x7fdd48157520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd47fff640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd480b3a20 0x7fdd48157520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd47fff640 1 -- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd480cfee0 msgr2=0x7fdd480aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd47fff640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd480cfee0 0x7fdd480aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd47fff640 1 -- 172.21.15.161:0/988870761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd380094d0 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd46ffd640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd480cfee0 0x7fdd480aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd47fff640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd4815d440 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000c7a0 tx=0x7fdd4000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd2ffff640 1 -- 172.21.15.161:0/988870761 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000a640 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd2ffff640 1 -- 172.21.15.161:0/988870761 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd4000a7a0 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd480ae460 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd2ffff640 1 -- 172.21.15.161:0/988870761 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd4000abd0 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.755+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd48157a60 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.756+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd480ae980 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.757+0000 7fdd2ffff640 1 -- 172.21.15.161:0/988870761 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd4000ad30 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.757+0000 7fdd2ffff640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd3406b810 0x7fdd3406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.757+0000 7fdd2ffff640 1 -- 172.21.15.161:0/988870761 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 5709+0+0 (secure 0 0 0) 0x7fdd400899f0 con 0x7fdd48145520 2024-03-23T14:36:07.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.757+0000 7fdd477fe640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd3406b810 0x7fdd3406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:07.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.758+0000 7fdd477fe640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd3406b810 0x7fdd3406dcd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd38036000 tx=0x7fdd3803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:07.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.761+0000 7fdd2ffff640 1 -- 172.21.15.161:0/988870761 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd480ae980 con 0x7fdd48145520 2024-03-23T14:36:07.967 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:07.972+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fdd480ae980 con 0x7fdd48145520 2024-03-23T14:36:08.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.951+0000 7fdd2ffff640 1 -- 172.21.15.161:0/988870761 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) ==== 118+0+0 (secure 0 0 0) 0x7fdd480ae980 con 0x7fdd48145520 2024-03-23T14:36:08.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd3406b810 msgr2=0x7fdd3406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd3406b810 0x7fdd3406dcd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fdd38036000 tx=0x7fdd3803b040 comp rx=0 tx=0).stop 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 msgr2=0x7fdd4815d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd4815d440 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fdd4000c7a0 tx=0x7fdd4000cc60 comp rx=0 tx=0).stop 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 shutdown_connections 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd3406b810 0x7fdd3406dcd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd48145520 0x7fdd4815d440 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd480cfee0 0x7fdd480aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 --2- 172.21.15.161:0/988870761 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd480b3a20 0x7fdd48157520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.954+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 >> 172.21.15.161:0/988870761 conn(0x7fdd4806b4a0 msgr2=0x7fdd480af580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.955+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 shutdown_connections 2024-03-23T14:36:08.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:08.955+0000 7fdd4defb640 1 -- 172.21.15.161:0/988870761 wait complete. 2024-03-23T14:36:08.964 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T14:36:08.964 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T14:36:09.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.077+0000 7f8edd993640 1 Processor -- start 2024-03-23T14:36:09.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.077+0000 7f8edd993640 1 -- start start 2024-03-23T14:36:09.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8edd993640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8ed8146d60 0x7f8ed80ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8edd993640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8ed8161c40 0x7f8ed81620f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8edd993640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed80721f0 0x7f8ed8072680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8edd993640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8ed8072bc0 con 0x7f8ed8161c40 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8edd993640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f8ed8162700 con 0x7f8ed8146d60 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8edd993640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed81629e0 con 0x7f8ed80721f0 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed67fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8ed8161c40 0x7f8ed81620f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed67fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8ed8161c40 0x7f8ed81620f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38318/0 (socket says 172.21.15.161:38318) 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed67fc640 1 -- 172.21.15.161:0/2383096329 learned_addr learned my addr 172.21.15.161:0/2383096329 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed6ffd640 1 --2- 172.21.15.161:0/2383096329 >> v2:172.21.15.183:3300/0 conn(0x7f8ed8146d60 0x7f8ed80ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed77fe640 1 --2- 172.21.15.161:0/2383096329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed80721f0 0x7f8ed8072680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed67fc640 1 -- 172.21.15.161:0/2383096329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed80721f0 msgr2=0x7f8ed8072680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:09.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed67fc640 1 --2- 172.21.15.161:0/2383096329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed80721f0 0x7f8ed8072680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed67fc640 1 -- 172.21.15.161:0/2383096329 >> v2:172.21.15.183:3300/0 conn(0x7f8ed8146d60 msgr2=0x7f8ed80ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:09.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed67fc640 1 --2- 172.21.15.161:0/2383096329 >> v2:172.21.15.183:3300/0 conn(0x7f8ed8146d60 0x7f8ed80ad8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.078+0000 7f8ed67fc640 1 -- 172.21.15.161:0/2383096329 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed80b43c0 con 0x7f8ed8161c40 2024-03-23T14:36:09.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.079+0000 7f8ed67fc640 1 --2- 172.21.15.161:0/2383096329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed8161c40 0x7f8ed81620f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8070bc0 tx=0x7f8ecc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4d2613d4c960f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:09.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.079+0000 7f8ed5ffb640 1 -- 172.21.15.161:0/2383096329 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc031870 con 0x7f8ed8161c40 2024-03-23T14:36:09.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.079+0000 7f8ed5ffb640 1 -- 172.21.15.161:0/2383096329 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ecc0319d0 con 0x7f8ed8161c40 2024-03-23T14:36:09.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.079+0000 7f8ed5ffb640 1 -- 172.21.15.161:0/2383096329 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc031d60 con 0x7f8ed8161c40 2024-03-23T14:36:09.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 -- 172.21.15.161:0/2383096329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed8161c40 msgr2=0x7f8ed81620f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:09.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 --2- 172.21.15.161:0/2383096329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed8161c40 0x7f8ed81620f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8070bc0 tx=0x7f8ecc02b910 comp rx=0 tx=0).stop 2024-03-23T14:36:09.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 -- 172.21.15.161:0/2383096329 shutdown_connections 2024-03-23T14:36:09.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 --2- 172.21.15.161:0/2383096329 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed80721f0 0x7f8ed8072680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 --2- 172.21.15.161:0/2383096329 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed8161c40 0x7f8ed81620f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 --2- 172.21.15.161:0/2383096329 >> v2:172.21.15.183:3300/0 conn(0x7f8ed8146d60 0x7f8ed80ad8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.079 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 -- 172.21.15.161:0/2383096329 >> 172.21.15.161:0/2383096329 conn(0x7f8ed806ce50 msgr2=0x7f8ed806d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 -- 172.21.15.161:0/2383096329 shutdown_connections 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.080+0000 7f8edd993640 1 -- 172.21.15.161:0/2383096329 wait complete. 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.081+0000 7f8edd993640 1 Processor -- start 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.081+0000 7f8edd993640 1 -- start start 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8edd993640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed80721f0 0x7f8ed8160fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8edd993640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed8146d60 0x7f8ed80cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8edd993640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ed8161c40 0x7f8ed80ccfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8edd993640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed8139120 con 0x7f8ed80721f0 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8edd993640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed80cd4e0 con 0x7f8ed8161c40 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8edd993640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8ed80cd770 con 0x7f8ed8146d60 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed80721f0 0x7f8ed8160fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed80721f0 0x7f8ed8160fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38334/0 (socket says 172.21.15.161:38334) 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 -- 172.21.15.161:0/493213884 learned_addr learned my addr 172.21.15.161:0/493213884 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:09.080 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed67fc640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed8146d60 0x7f8ed80cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed77fe640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ed8161c40 0x7f8ed80ccfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 -- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed8146d60 msgr2=0x7f8ed80cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed8146d60 0x7f8ed80cca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 -- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ed8161c40 msgr2=0x7f8ed80ccfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ed8161c40 0x7f8ed80ccfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 -- 172.21.15.161:0/493213884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ecc0094d0 con 0x7f8ed80721f0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ed6ffd640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed80721f0 0x7f8ed8160fb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800e9b0 tx=0x7f8ec800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ebffff640 1 -- 172.21.15.161:0/493213884 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8009670 con 0x7f8ed80721f0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ebffff640 1 -- 172.21.15.161:0/493213884 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ec800c9a0 con 0x7f8ed80721f0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8ebffff640 1 -- 172.21.15.161:0/493213884 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800cd50 con 0x7f8ed80721f0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.082+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed80cda50 con 0x7f8ed80721f0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.083+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed80cdd30 con 0x7f8ed80721f0 2024-03-23T14:36:09.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.084+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ed8139390 con 0x7f8ed80721f0 2024-03-23T14:36:09.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.084+0000 7f8ebffff640 1 -- 172.21.15.161:0/493213884 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8ec8013470 con 0x7f8ed80721f0 2024-03-23T14:36:09.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.084+0000 7f8ebffff640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8ec406b810 0x7f8ec406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:09.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.084+0000 7f8ebffff640 1 -- 172.21.15.161:0/493213884 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f8ec808a2e0 con 0x7f8ed80721f0 2024-03-23T14:36:09.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.084+0000 7f8ed67fc640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8ec406b810 0x7f8ec406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:09.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.085+0000 7f8ed67fc640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8ec406b810 0x7f8ec406dcd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc036000 tx=0x7f8ecc03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:09.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.089+0000 7f8ebffff640 1 -- 172.21.15.161:0/493213884 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8ec8054e40 con 0x7f8ed80721f0 2024-03-23T14:36:09.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.296+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7f8ed8072710 con 0x7f8ed80721f0 2024-03-23T14:36:09.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.952+0000 7f8ebffff640 1 -- 172.21.15.161:0/493213884 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v15) ==== 122+0+0 (secure 0 0 0) 0x7f8ec8058500 con 0x7f8ed80721f0 2024-03-23T14:36:09.950 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8ec406b810 msgr2=0x7f8ec406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8ec406b810 0x7f8ec406dcd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc036000 tx=0x7f8ecc03c040 comp rx=0 tx=0).stop 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed80721f0 msgr2=0x7f8ed8160fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed80721f0 0x7f8ed8160fb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800e9b0 tx=0x7f8ec800ee70 comp rx=0 tx=0).stop 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 shutdown_connections 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8ec406b810 0x7f8ec406dcd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8ed8161c40 0x7f8ed80ccfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8ed8146d60 0x7f8ed80cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 --2- 172.21.15.161:0/493213884 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8ed80721f0 0x7f8ed8160fb0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 >> 172.21.15.161:0/493213884 conn(0x7f8ed806ce50 msgr2=0x7f8ed80aed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 shutdown_connections 2024-03-23T14:36:09.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:09.954+0000 7f8edd993640 1 -- 172.21.15.161:0/493213884 wait complete. 2024-03-23T14:36:09.963 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T14:36:09.963 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:36:10.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.070+0000 7f746c181640 1 Processor -- start 2024-03-23T14:36:10.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.071+0000 7f746c181640 1 -- start start 2024-03-23T14:36:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.071+0000 7f746c181640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f746415e400 0x7f74640cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.071+0000 7f746c181640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 0x7f74640b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.071+0000 7f746c181640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f74640d0460 con 0x7f74640cf9d0 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.071+0000 7f746c181640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f74640d1050 con 0x7f746415e400 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.071+0000 7f746c181640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f74640d06b0 con 0x7f74640d2b40 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.071+0000 7f7469f1d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f746415e400 0x7f74640cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.072+0000 7f7469f1d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f746415e400 0x7f74640cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38348/0 (socket says 172.21.15.161:38348) 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.072+0000 7f7469f1d640 1 -- 172.21.15.161:0/3847209686 learned_addr learned my addr 172.21.15.161:0/3847209686 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.072+0000 7f746a71e640 1 --2- 172.21.15.161:0/3847209686 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 0x7f74640b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.072+0000 7f7469f1d640 1 -- 172.21.15.161:0/3847209686 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 msgr2=0x7f74640b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.072+0000 7f7469f1d640 1 --2- 172.21.15.161:0/3847209686 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 0x7f74640b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.072+0000 7f7469f1d640 1 -- 172.21.15.161:0/3847209686 >> v1:172.21.15.183:6789/0 conn(0x7f74640cf9d0 legacy=0x7f74640d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.072+0000 7f7469f1d640 1 -- 172.21.15.161:0/3847209686 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74640d0990 con 0x7f746415e400 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.072+0000 7f7469f1d640 1 --2- 172.21.15.161:0/3847209686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f74640cf490 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7460009870 tx=0x7f746002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe4305734b5dfc7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f7468f1b640 1 -- 172.21.15.161:0/3847209686 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460030840 con 0x7f746415e400 2024-03-23T14:36:10.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f7468f1b640 1 -- 172.21.15.161:0/3847209686 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74600309a0 con 0x7f746415e400 2024-03-23T14:36:10.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f7468f1b640 1 -- 172.21.15.161:0/3847209686 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7460030d30 con 0x7f746415e400 2024-03-23T14:36:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f746c181640 1 -- 172.21.15.161:0/3847209686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 msgr2=0x7f74640cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f746c181640 1 --2- 172.21.15.161:0/3847209686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f74640cf490 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f7460009870 tx=0x7f746002b980 comp rx=0 tx=0).stop 2024-03-23T14:36:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f746c181640 1 -- 172.21.15.161:0/3847209686 shutdown_connections 2024-03-23T14:36:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f746c181640 1 --2- 172.21.15.161:0/3847209686 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 0x7f74640b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f746c181640 1 --2- 172.21.15.161:0/3847209686 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f74640cf490 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f746c181640 1 -- 172.21.15.161:0/3847209686 >> 172.21.15.161:0/3847209686 conn(0x7f746409d190 msgr2=0x7f74640d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f746c181640 1 -- 172.21.15.161:0/3847209686 shutdown_connections 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.073+0000 7f746c181640 1 -- 172.21.15.161:0/3847209686 wait complete. 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.074+0000 7f746c181640 1 Processor -- start 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.074+0000 7f746c181640 1 -- start start 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746c181640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74640cf9d0 0x7f74641f7e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746c181640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 0x7f746414e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746c181640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f746415e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746c181640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f746415f6f0 con 0x7f746415e400 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746c181640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f746415f130 con 0x7f74640cf9d0 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746c181640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f746415f410 con 0x7f74640d2b40 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746a71e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f746415e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746a71e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f746415e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38350/0 (socket says 172.21.15.161:38350) 2024-03-23T14:36:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746a71e640 1 -- 172.21.15.161:0/64905702 learned_addr learned my addr 172.21.15.161:0/64905702 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f7469f1d640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74640cf9d0 0x7f74641f7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.075+0000 7f746971c640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 0x7f746414e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f746a71e640 1 -- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 msgr2=0x7f746414e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f746a71e640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 0x7f746414e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f746a71e640 1 -- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74640cf9d0 msgr2=0x7f74641f7e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f746a71e640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74640cf9d0 0x7f74641f7e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f746a71e640 1 -- 172.21.15.161:0/64905702 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74600094d0 con 0x7f746415e400 2024-03-23T14:36:10.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f746a71e640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f746415e7e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f745c00efc0 tx=0x7f745c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f745a7fc640 1 -- 172.21.15.161:0/64905702 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c014480 con 0x7f746415e400 2024-03-23T14:36:10.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f745a7fc640 1 -- 172.21.15.161:0/64905702 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f745c00f040 con 0x7f746415e400 2024-03-23T14:36:10.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f745a7fc640 1 -- 172.21.15.161:0/64905702 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745c014820 con 0x7f746415e400 2024-03-23T14:36:10.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74641f7620 con 0x7f746415e400 2024-03-23T14:36:10.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.076+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74641f8360 con 0x7f746415e400 2024-03-23T14:36:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.077+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f74641f7480 con 0x7f746415e400 2024-03-23T14:36:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.077+0000 7f745a7fc640 1 -- 172.21.15.161:0/64905702 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f745c014980 con 0x7f746415e400 2024-03-23T14:36:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.078+0000 7f745a7fc640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f744806b810 0x7f744806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.078+0000 7f745a7fc640 1 -- 172.21.15.161:0/64905702 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f745c08ac10 con 0x7f746415e400 2024-03-23T14:36:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.078+0000 7f7469f1d640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f744806b810 0x7f744806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.078+0000 7f7469f1d640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f744806b810 0x7f744806dcd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7460036000 tx=0x7f7460008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.082+0000 7f745a7fc640 1 -- 172.21.15.161:0/64905702 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f74641f7480 con 0x7f746415e400 2024-03-23T14:36:10.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.288+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f74640d1450 con 0x7f746415e400 2024-03-23T14:36:10.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.289+0000 7f745a7fc640 1 -- 172.21.15.161:0/64905702 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5180 (secure 0 0 0) 0x7f74640d1450 con 0x7f746415e400 2024-03-23T14:36:10.285 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:10.285 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/626265302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":626265302},{"type":"v1","addr":"172.21.15.183:6837","nonce":626265302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5220,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/208344896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":208344896},{"type":"v1","addr":"172.21.15.161:6833","nonce":208344896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/82303731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":82303731},{"type":"v1","addr":"172.21.15.161:6835","nonce":82303731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:06.267736+0000","modified":"2024-03-23T14:36:09.297699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5235":{"gid":5235,"name":"d","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6836/2372368263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2372368263},{"type":"v1","addr":"172.21.15.183:6836","nonce":2372368263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5235,"qdb_cluster":[5235]},"id":2}]} 2024-03-23T14:36:10.285 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 15 2024-03-23T14:36:10.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f744806b810 msgr2=0x7f744806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f744806b810 0x7f744806dcd0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7460036000 tx=0x7f7460008030 comp rx=0 tx=0).stop 2024-03-23T14:36:10.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 msgr2=0x7f746415e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f746415e7e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f745c00efc0 tx=0x7f745c00c490 comp rx=0 tx=0).stop 2024-03-23T14:36:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 shutdown_connections 2024-03-23T14:36:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f744806b810 0x7f744806dcd0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f746415e400 0x7f746415e7e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74640d2b40 0x7f746414e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 --2- 172.21.15.161:0/64905702 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f74640cf9d0 0x7f74641f7e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 >> 172.21.15.161:0/64905702 conn(0x7f746409d190 msgr2=0x7f74640b0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 shutdown_connections 2024-03-23T14:36:10.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.291+0000 7f746c181640 1 -- 172.21.15.161:0/64905702 wait complete. 2024-03-23T14:36:10.298 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:10.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.402+0000 7f40d7b79640 1 Processor -- start 2024-03-23T14:36:10.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.402+0000 7f40d7b79640 1 -- start start 2024-03-23T14:36:10.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d7b79640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f40d00b3200 0x7f40d00b4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d7b79640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f40d006e590 0x7f40d006e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d7b79640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f40d0144ae0 con 0x7f40d0144700 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d7b79640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f40d0070ec0 con 0x7f40d00b3200 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d7b79640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f40d006eee0 con 0x7f40d006e590 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d5114640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f40d00b3200 0x7f40d00b4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d5114640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f40d00b3200 0x7f40d00b4770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38358/0 (socket says 172.21.15.161:38358) 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d5114640 1 -- 172.21.15.161:0/3722959424 learned_addr learned my addr 172.21.15.161:0/3722959424 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.403+0000 7f40d6116640 1 --2- 172.21.15.161:0/3722959424 >> v2:172.21.15.183:3300/0 conn(0x7f40d006e590 0x7f40d006e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.404+0000 7f40d5114640 1 -- 172.21.15.161:0/3722959424 >> v2:172.21.15.183:3300/0 conn(0x7f40d006e590 msgr2=0x7f40d006e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.404+0000 7f40d5114640 1 --2- 172.21.15.161:0/3722959424 >> v2:172.21.15.183:3300/0 conn(0x7f40d006e590 0x7f40d006e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.404+0000 7f40d5114640 1 -- 172.21.15.161:0/3722959424 >> v1:172.21.15.183:6789/0 conn(0x7f40d0144700 legacy=0x7f40d00b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.404+0000 7f40d5114640 1 -- 172.21.15.161:0/3722959424 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40d00b36d0 con 0x7f40d00b3200 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.404+0000 7f40d4913640 1 -- 172.21.15.161:0/3722959424 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3818190002 0 0) 0x7f40d0144ae0 con 0x7f40d0144700 2024-03-23T14:36:10.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.404+0000 7f40d6116640 1 --2- 172.21.15.161:0/3722959424 >> v2:172.21.15.183:3300/0 conn(0x7f40d006e590 0x7f40d006e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.404+0000 7f40d5114640 1 --2- 172.21.15.161:0/3722959424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d00b3200 0x7f40d00b4770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f40d006ccf0 tx=0x7f40c002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a00270898053fed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.404+0000 7f40d4913640 1 -- 172.21.15.161:0/3722959424 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0030860 con 0x7f40d00b3200 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d4913640 1 -- 172.21.15.161:0/3722959424 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c00309c0 con 0x7f40d00b3200 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d4913640 1 -- 172.21.15.161:0/3722959424 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c0030d50 con 0x7f40d00b3200 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d7b79640 1 -- 172.21.15.161:0/3722959424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d00b3200 msgr2=0x7f40d00b4770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d7b79640 1 --2- 172.21.15.161:0/3722959424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d00b3200 0x7f40d00b4770 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f40d006ccf0 tx=0x7f40c002b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d7b79640 1 -- 172.21.15.161:0/3722959424 shutdown_connections 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d7b79640 1 --2- 172.21.15.161:0/3722959424 >> v2:172.21.15.183:3300/0 conn(0x7f40d006e590 0x7f40d006e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d7b79640 1 --2- 172.21.15.161:0/3722959424 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d00b3200 0x7f40d00b4770 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d7b79640 1 -- 172.21.15.161:0/3722959424 >> 172.21.15.161:0/3722959424 conn(0x7f40d00b17f0 msgr2=0x7f40d00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:10.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d7b79640 1 -- 172.21.15.161:0/3722959424 shutdown_connections 2024-03-23T14:36:10.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.405+0000 7f40d7b79640 1 -- 172.21.15.161:0/3722959424 wait complete. 2024-03-23T14:36:10.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.406+0000 7f40d7b79640 1 Processor -- start 2024-03-23T14:36:10.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.406+0000 7f40d7b79640 1 -- start start 2024-03-23T14:36:10.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d7b79640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40d006e590 0x7f40d00c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d7b79640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40d00b3200 0x7f40d013cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d7b79640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d0144700 0x7f40d006d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d7b79640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f40d013db80 con 0x7f40d0144700 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d7b79640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f40d015a6e0 con 0x7f40d006e590 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d7b79640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f40d015a9c0 con 0x7f40d00b3200 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d6116640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d0144700 0x7f40d006d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d6116640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d0144700 0x7f40d006d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38368/0 (socket says 172.21.15.161:38368) 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d6116640 1 -- 172.21.15.161:0/1868669 learned_addr learned my addr 172.21.15.161:0/1868669 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d5915640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40d006e590 0x7f40d00c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d5114640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40d00b3200 0x7f40d013cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d6116640 1 -- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40d00b3200 msgr2=0x7f40d013cf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d6116640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40d00b3200 0x7f40d013cf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d6116640 1 -- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40d006e590 msgr2=0x7f40d00c4a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d6116640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40d006e590 0x7f40d00c4a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.407+0000 7f40d6116640 1 -- 172.21.15.161:0/1868669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40c800ba70 con 0x7f40d0144700 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.408+0000 7f40d6116640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d0144700 0x7f40d006d4f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f40c8010f70 tx=0x7f40c800e490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.408+0000 7f40c67fc640 1 -- 172.21.15.161:0/1868669 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c800bcc0 con 0x7f40d0144700 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.408+0000 7f40c67fc640 1 -- 172.21.15.161:0/1868669 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40c8011040 con 0x7f40d0144700 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.408+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40c00094d0 con 0x7f40d0144700 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.408+0000 7f40c67fc640 1 -- 172.21.15.161:0/1868669 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40c8003990 con 0x7f40d0144700 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.408+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40d015af80 con 0x7f40d0144700 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.409+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f40d00735e0 con 0x7f40d0144700 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.409+0000 7f40c67fc640 1 -- 172.21.15.161:0/1868669 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f40c8016480 con 0x7f40d0144700 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.410+0000 7f40c67fc640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f40b406b810 0x7f40b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.410+0000 7f40c67fc640 1 -- 172.21.15.161:0/1868669 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f40c808cab0 con 0x7f40d0144700 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.410+0000 7f40d5915640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f40b406b810 0x7f40b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.410+0000 7f40d5915640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f40b406b810 0x7f40b406dcd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f40cc000f60 tx=0x7f40cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.414+0000 7f40c67fc640 1 -- 172.21.15.161:0/1868669 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f40c80574f0 con 0x7f40d0144700 2024-03-23T14:36:10.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.588+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f40d004f550 con 0x7f40d0144700 2024-03-23T14:36:10.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.589+0000 7f40c67fc640 1 -- 172.21.15.161:0/1868669 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) ==== 74+0+16705 (secure 0 0 0) 0x7f40c801a020 con 0x7f40d0144700 2024-03-23T14:36:10.585 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:10.586 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":33,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:07.925548+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":5,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:10.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f40b406b810 msgr2=0x7f40b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f40b406b810 0x7f40b406dcd0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f40cc000f60 tx=0x7f40cc009040 comp rx=0 tx=0).stop 2024-03-23T14:36:10.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d0144700 msgr2=0x7f40d006d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d0144700 0x7f40d006d4f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f40c8010f70 tx=0x7f40c800e490 comp rx=0 tx=0).stop 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 shutdown_connections 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f40b406b810 0x7f40b406dcd0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f40d0144700 0x7f40d006d4f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f40d00b3200 0x7f40d013cf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 --2- 172.21.15.161:0/1868669 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f40d006e590 0x7f40d00c4a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 >> 172.21.15.161:0/1868669 conn(0x7f40d00b17f0 msgr2=0x7f40d01430a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 shutdown_connections 2024-03-23T14:36:10.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.591+0000 7f40d7b79640 1 -- 172.21.15.161:0/1868669 wait complete. 2024-03-23T14:36:10.598 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T14:36:10.598 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:36:10.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.703+0000 7f8b30f8a640 1 Processor -- start 2024-03-23T14:36:10.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.704+0000 7f8b30f8a640 1 -- start start 2024-03-23T14:36:10.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.704+0000 7f8b30f8a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b2c15e400 0x7f8b2c0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.704+0000 7f8b30f8a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c0d0460 con 0x7f8b2c0d2b40 2024-03-23T14:36:10.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.704+0000 7f8b30f8a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b2c0d06b0 con 0x7f8b2c0cf9d0 2024-03-23T14:36:10.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.704+0000 7f8b30f8a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8b2c0d1050 con 0x7f8b2c15e400 2024-03-23T14:36:10.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.704+0000 7f8b2a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8b2c15e400 0x7f8b2c0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.704+0000 7f8b2ad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8b2c0d2b40 0x7f8b2c0b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45444/0 (socket says 172.21.15.161:45444) 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.704+0000 7f8b2ad9d640 1 -- 172.21.15.161:0/3096150861 learned_addr learned my addr 172.21.15.161:0/3096150861 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.705+0000 7f8b2a59c640 1 -- 172.21.15.161:0/3096150861 >> v1:172.21.15.183:6789/0 conn(0x7f8b2c0cf9d0 legacy=0x7f8b2c0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.705+0000 7f8b2a59c640 1 -- 172.21.15.161:0/3096150861 >> v1:172.21.15.161:6789/0 conn(0x7f8b2c0d2b40 legacy=0x7f8b2c0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.705+0000 7f8b29d9b640 1 -- 172.21.15.161:0/3096150861 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607929213 0 0) 0x7f8b2c0d06b0 con 0x7f8b2c0cf9d0 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.705+0000 7f8b2a59c640 1 -- 172.21.15.161:0/3096150861 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2c0d0990 con 0x7f8b2c15e400 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.705+0000 7f8b2a59c640 1 --2- 172.21.15.161:0/3096150861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c0cf490 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c0097a0 tx=0x7f8b1c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1801464f97595bc1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b29d9b640 1 -- 172.21.15.161:0/3096150861 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c030710 con 0x7f8b2c15e400 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b29d9b640 1 -- 172.21.15.161:0/3096150861 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b1c02e050 con 0x7f8b2c15e400 2024-03-23T14:36:10.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b29d9b640 1 -- 172.21.15.161:0/3096150861 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c030a90 con 0x7f8b2c15e400 2024-03-23T14:36:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3096150861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 msgr2=0x7f8b2c0cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b30f8a640 1 --2- 172.21.15.161:0/3096150861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c0cf490 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c0097a0 tx=0x7f8b1c02b790 comp rx=0 tx=0).stop 2024-03-23T14:36:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3096150861 shutdown_connections 2024-03-23T14:36:10.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b30f8a640 1 --2- 172.21.15.161:0/3096150861 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c0cf490 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3096150861 >> 172.21.15.161:0/3096150861 conn(0x7f8b2c09d190 msgr2=0x7f8b2c0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3096150861 shutdown_connections 2024-03-23T14:36:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.706+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3096150861 wait complete. 2024-03-23T14:36:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.707+0000 7f8b30f8a640 1 Processor -- start 2024-03-23T14:36:10.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b30f8a640 1 -- start start 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b30f8a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b2c0cf9d0 0x7f8b2c1f4b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b30f8a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b2c0d2b40 0x7f8b2c0c83d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b30f8a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c15c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b30f8a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2c0c7fb0 con 0x7f8b2c15e400 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b30f8a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2c1f50c0 con 0x7f8b2c0cf9d0 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b30f8a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8b2c0c8910 con 0x7f8b2c0d2b40 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b2ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c15c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b2ad9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c15c3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36440/0 (socket says 172.21.15.161:36440) 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b2ad9d640 1 -- 172.21.15.161:0/3342031538 learned_addr learned my addr 172.21.15.161:0/3342031538 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:10.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b2a59c640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b2c0cf9d0 0x7f8b2c1f4b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.708+0000 7f8b23fff640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b2c0d2b40 0x7f8b2c0c83d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b2ad9d640 1 -- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b2c0d2b40 msgr2=0x7f8b2c0c83d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b2ad9d640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b2c0d2b40 0x7f8b2c0c83d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b2ad9d640 1 -- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b2c0cf9d0 msgr2=0x7f8b2c1f4b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b2ad9d640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b2c0cf9d0 0x7f8b2c1f4b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b2ad9d640 1 -- 172.21.15.161:0/3342031538 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b1c009400 con 0x7f8b2c15e400 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b2ad9d640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c15c3b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400caf0 tx=0x7f8b240020c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b22ffd640 1 -- 172.21.15.161:0/3342031538 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2400a700 con 0x7f8b2c15e400 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b22ffd640 1 -- 172.21.15.161:0/3342031538 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8b2400a860 con 0x7f8b2c15e400 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b22ffd640 1 -- 172.21.15.161:0/3342031538 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b2400ac10 con 0x7f8b2c15e400 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2c0c7980 con 0x7f8b2c15e400 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.709+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b2c0c9ac0 con 0x7f8b2c15e400 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.710+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b2c068680 con 0x7f8b2c15e400 2024-03-23T14:36:10.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.710+0000 7f8b22ffd640 1 -- 172.21.15.161:0/3342031538 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8b240070d0 con 0x7f8b2c15e400 2024-03-23T14:36:10.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.711+0000 7f8b22ffd640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b0c06b810 0x7f8b0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:10.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.711+0000 7f8b22ffd640 1 -- 172.21.15.161:0/3342031538 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f8b24089b60 con 0x7f8b2c15e400 2024-03-23T14:36:10.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.711+0000 7f8b2a59c640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b0c06b810 0x7f8b0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:10.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.711+0000 7f8b2a59c640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b0c06b810 0x7f8b0c06dcd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c034000 tx=0x7f8b1c02f000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:10.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.715+0000 7f8b22ffd640 1 -- 172.21.15.161:0/3342031538 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8b24054630 con 0x7f8b2c15e400 2024-03-23T14:36:10.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.881+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8b2c0c80f0 con 0x7f8b0c06b810 2024-03-23T14:36:10.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.887+0000 7f8b22ffd640 1 -- 172.21.15.161:0/3342031538 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239826 (secure 0 0 0) 0x7f8b2c0c80f0 con 0x7f8b0c06b810 2024-03-23T14:36:10.886 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:10.887 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b0c06b810 msgr2=0x7f8b0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b0c06b810 0x7f8b0c06dcd0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c034000 tx=0x7f8b1c02f000 comp rx=0 tx=0).stop 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 msgr2=0x7f8b2c15c3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c15c3b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8b2400caf0 tx=0x7f8b240020c0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 shutdown_connections 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8b0c06b810 0x7f8b0c06dcd0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8b2c15e400 0x7f8b2c15c3b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8b2c0d2b40 0x7f8b2c0c83d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 --2- 172.21.15.161:0/3342031538 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8b2c0cf9d0 0x7f8b2c1f4b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 >> 172.21.15.161:0/3342031538 conn(0x7f8b2c09d190 msgr2=0x7f8b2c0b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 shutdown_connections 2024-03-23T14:36:10.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:10.890+0000 7f8b30f8a640 1 -- 172.21.15.161:0/3342031538 wait complete. 2024-03-23T14:36:10.897 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-03-23T14:36:10.484585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219920,"kb_used_data":3748,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754800,"statfs":{"total":773094113280,"available":772868915200,"internally_reserved":0,"allocated":3837952,"data_stored":2181477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.568230"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921146+0000","last_change":"2024-03-23T14:36:04.921146+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921146+0000","last_undegraded":"2024-03-23T14:36:04.921146+0000","last_fullsized":"2024-03-23T14:36:04.921146+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919587+0000","last_change":"2024-03-23T14:36:04.919587+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919587+0000","last_undegraded":"2024-03-23T14:36:04.919587+0000","last_fullsized":"2024-03-23T14:36:04.919587+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922002+0000","last_change":"2024-03-23T14:36:04.922002+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922002+0000","last_undegraded":"2024-03-23T14:36:04.922002+0000","last_fullsized":"2024-03-23T14:36:04.922002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920050+0000","last_change":"2024-03-23T14:36:04.920050+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920050+0000","last_undegraded":"2024-03-23T14:36:04.920050+0000","last_fullsized":"2024-03-23T14:36:04.920050+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919319+0000","last_change":"2024-03-23T14:36:04.919319+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919319+0000","last_undegraded":"2024-03-23T14:36:04.919319+0000","last_fullsized":"2024-03-23T14:36:04.919319+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918554+0000","last_change":"2024-03-23T14:36:04.918554+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918554+0000","last_undegraded":"2024-03-23T14:36:04.918554+0000","last_fullsized":"2024-03-23T14:36:04.918554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921218+0000","last_change":"2024-03-23T14:36:04.921218+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921218+0000","last_undegraded":"2024-03-23T14:36:04.921218+0000","last_fullsized":"2024-03-23T14:36:04.921218+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919010+0000","last_change":"2024-03-23T14:36:04.919010+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919010+0000","last_undegraded":"2024-03-23T14:36:04.919010+0000","last_fullsized":"2024-03-23T14:36:04.919010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920263+0000","last_change":"2024-03-23T14:36:04.920263+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920263+0000","last_undegraded":"2024-03-23T14:36:04.920263+0000","last_fullsized":"2024-03-23T14:36:04.920263+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919149+0000","last_change":"2024-03-23T14:36:04.919149+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919149+0000","last_undegraded":"2024-03-23T14:36:04.919149+0000","last_fullsized":"2024-03-23T14:36:04.919149+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921107+0000","last_change":"2024-03-23T14:36:04.921107+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921107+0000","last_undegraded":"2024-03-23T14:36:04.921107+0000","last_fullsized":"2024-03-23T14:36:04.921107+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920278+0000","last_change":"2024-03-23T14:36:04.920278+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920278+0000","last_undegraded":"2024-03-23T14:36:04.920278+0000","last_fullsized":"2024-03-23T14:36:04.920278+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918667+0000","last_change":"2024-03-23T14:36:04.918667+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918667+0000","last_undegraded":"2024-03-23T14:36:04.918667+0000","last_fullsized":"2024-03-23T14:36:04.918667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918094+0000","last_change":"2024-03-23T14:36:04.918094+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918094+0000","last_undegraded":"2024-03-23T14:36:04.918094+0000","last_fullsized":"2024-03-23T14:36:04.918094+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921589+0000","last_change":"2024-03-23T14:36:04.921589+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921589+0000","last_undegraded":"2024-03-23T14:36:04.921589+0000","last_fullsized":"2024-03-23T14:36:04.921589+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920831+0000","last_change":"2024-03-23T14:36:04.920831+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920831+0000","last_undegraded":"2024-03-23T14:36:04.920831+0000","last_fullsized":"2024-03-23T14:36:04.920831+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918060+0000","last_change":"2024-03-23T14:36:04.918060+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918060+0000","last_undegraded":"2024-03-23T14:36:04.918060+0000","last_fullsized":"2024-03-23T14:36:04.918060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919674+0000","last_change":"2024-03-23T14:36:04.919674+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919674+0000","last_undegraded":"2024-03-23T14:36:04.919674+0000","last_fullsized":"2024-03-23T14:36:04.919674+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919612+0000","last_change":"2024-03-23T14:36:04.919612+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919612+0000","last_undegraded":"2024-03-23T14:36:04.919612+0000","last_fullsized":"2024-03-23T14:36:04.919612+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918594+0000","last_change":"2024-03-23T14:36:04.918594+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918594+0000","last_undegraded":"2024-03-23T14:36:04.918594+0000","last_fullsized":"2024-03-23T14:36:04.918594+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918956+0000","last_change":"2024-03-23T14:36:04.918956+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918956+0000","last_undegraded":"2024-03-23T14:36:04.918956+0000","last_fullsized":"2024-03-23T14:36:04.918956+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921160+0000","last_change":"2024-03-23T14:36:04.921160+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921160+0000","last_undegraded":"2024-03-23T14:36:04.921160+0000","last_fullsized":"2024-03-23T14:36:04.921160+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920128+0000","last_change":"2024-03-23T14:36:04.920128+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920128+0000","last_undegraded":"2024-03-23T14:36:04.920128+0000","last_fullsized":"2024-03-23T14:36:04.920128+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922010+0000","last_change":"2024-03-23T14:36:04.922010+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922010+0000","last_undegraded":"2024-03-23T14:36:04.922010+0000","last_fullsized":"2024-03-23T14:36:04.922010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917607+0000","last_change":"2024-03-23T14:36:04.917607+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917607+0000","last_undegraded":"2024-03-23T14:36:04.917607+0000","last_fullsized":"2024-03-23T14:36:04.917607+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918696+0000","last_change":"2024-03-23T14:36:04.918696+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918696+0000","last_undegraded":"2024-03-23T14:36:04.918696+0000","last_fullsized":"2024-03-23T14:36:04.918696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917667+0000","last_change":"2024-03-23T14:36:04.917667+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917667+0000","last_undegraded":"2024-03-23T14:36:04.917667+0000","last_fullsized":"2024-03-23T14:36:04.917667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918505+0000","last_change":"2024-03-23T14:36:04.918505+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918505+0000","last_undegraded":"2024-03-23T14:36:04.918505+0000","last_fullsized":"2024-03-23T14:36:04.918505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917659+0000","last_change":"2024-03-23T14:36:04.917659+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917659+0000","last_undegraded":"2024-03-23T14:36:04.917659+0000","last_fullsized":"2024-03-23T14:36:04.917659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918561+0000","last_change":"2024-03-23T14:36:04.918561+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918561+0000","last_undegraded":"2024-03-23T14:36:04.918561+0000","last_fullsized":"2024-03-23T14:36:04.918561+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922105+0000","last_change":"2024-03-23T14:36:04.922105+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922105+0000","last_undegraded":"2024-03-23T14:36:04.922105+0000","last_fullsized":"2024-03-23T14:36:04.922105+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921452+0000","last_change":"2024-03-23T14:36:04.921452+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921452+0000","last_undegraded":"2024-03-23T14:36:04.921452+0000","last_fullsized":"2024-03-23T14:36:04.921452+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-23T14:36:04.922822+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:36:04.922822+0000","last_peered":"2024-03-23T14:36:04.922822+0000","last_clean":"2024-03-23T14:36:04.922822+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:36:04.922822+0000","last_undegraded":"2024-03-23T14:36:04.922822+0000","last_fullsized":"2024-03-23T14:36:04.922822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":917504,"data_stored":711825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":121457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":121457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:36:10.901 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:36:11.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.042+0000 7f05b413e640 1 Processor -- start 2024-03-23T14:36:11.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.043+0000 7f05b413e640 1 -- start start 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.043+0000 7f05b413e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f05ac0d1470 0x7f05ac0d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.043+0000 7f05b413e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05ac0d0460 con 0x7f05ac15e400 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.043+0000 7f05b413e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05ac0d06b0 con 0x7f05ac0d2970 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.043+0000 7f05b413e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f05ac0d1050 con 0x7f05ac0d1470 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b1eda640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f05ac15e400 0x7f05ac0cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45450/0 (socket says 172.21.15.161:45450) 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b1eda640 1 -- 172.21.15.161:0/697916385 learned_addr learned my addr 172.21.15.161:0/697916385 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b16d9640 1 --2- 172.21.15.161:0/697916385 >> v2:172.21.15.183:3300/0 conn(0x7f05ac0d1470 0x7f05ac0d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b0ed8640 1 -- 172.21.15.161:0/697916385 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2648762359 0 0) 0x7f05ac0d0460 con 0x7f05ac15e400 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b0ed8640 1 -- 172.21.15.161:0/697916385 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0598003520 con 0x7f05ac15e400 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b16d9640 1 -- 172.21.15.161:0/697916385 >> v1:172.21.15.183:6789/0 conn(0x7f05ac0d2970 legacy=0x7f05ac0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b0ed8640 1 -- 172.21.15.161:0/697916385 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3408233007 0 0) 0x7f05ac0d06b0 con 0x7f05ac0d2970 2024-03-23T14:36:11.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b16d9640 1 -- 172.21.15.161:0/697916385 >> v1:172.21.15.161:6789/0 conn(0x7f05ac15e400 legacy=0x7f05ac0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.044+0000 7f05b16d9640 1 -- 172.21.15.161:0/697916385 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05ac0d0990 con 0x7f05ac0d1470 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b16d9640 1 --2- 172.21.15.161:0/697916385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 0x7f05ac0d2430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f05a80097a0 tx=0x7f05a802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6e7dc996ce86443b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b0ed8640 1 -- 172.21.15.161:0/697916385 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8030610 con 0x7f05ac0d1470 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b0ed8640 1 -- 172.21.15.161:0/697916385 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a8030770 con 0x7f05ac0d1470 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b0ed8640 1 -- 172.21.15.161:0/697916385 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8030ae0 con 0x7f05ac0d1470 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b413e640 1 -- 172.21.15.161:0/697916385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 msgr2=0x7f05ac0d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b413e640 1 --2- 172.21.15.161:0/697916385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 0x7f05ac0d2430 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f05a80097a0 tx=0x7f05a802b960 comp rx=0 tx=0).stop 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b413e640 1 -- 172.21.15.161:0/697916385 shutdown_connections 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b413e640 1 --2- 172.21.15.161:0/697916385 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 0x7f05ac0d2430 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b413e640 1 -- 172.21.15.161:0/697916385 >> 172.21.15.161:0/697916385 conn(0x7f05ac09d190 msgr2=0x7f05ac0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:11.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.045+0000 7f05b413e640 1 -- 172.21.15.161:0/697916385 shutdown_connections 2024-03-23T14:36:11.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.046+0000 7f05b413e640 1 -- 172.21.15.161:0/697916385 wait complete. 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b413e640 1 Processor -- start 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b413e640 1 -- start start 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b413e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 0x7f05ac0b42b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b413e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05ac0d2970 0x7f05ac0b47f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b413e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05ac15e400 0x7f05ac0c8df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b413e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac0ca730 con 0x7f05ac0d2970 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b413e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f05ac0c9b20 con 0x7f05ac0d1470 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b413e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f05ac0c9e00 con 0x7f05ac15e400 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b16d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05ac0d2970 0x7f05ac0b47f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b16d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05ac0d2970 0x7f05ac0b47f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36448/0 (socket says 172.21.15.161:36448) 2024-03-23T14:36:11.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b16d9640 1 -- 172.21.15.161:0/2475339887 learned_addr learned my addr 172.21.15.161:0/2475339887 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.047+0000 7f05b1eda640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 0x7f05ac0b42b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b26db640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05ac15e400 0x7f05ac0c8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b16d9640 1 -- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05ac15e400 msgr2=0x7f05ac0c8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b16d9640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05ac15e400 0x7f05ac0c8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b16d9640 1 -- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 msgr2=0x7f05ac0b42b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b16d9640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 0x7f05ac0b42b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b16d9640 1 -- 172.21.15.161:0/2475339887 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a8009400 con 0x7f05ac0d2970 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b16d9640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05ac0d2970 0x7f05ac0b47f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f05a8000c00 tx=0x7f05a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05a27fc640 1 -- 172.21.15.161:0/2475339887 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a803e070 con 0x7f05ac0d2970 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05a27fc640 1 -- 172.21.15.161:0/2475339887 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a8030e10 con 0x7f05ac0d2970 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05ac0ca0e0 con 0x7f05ac0d2970 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05a27fc640 1 -- 172.21.15.161:0/2475339887 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8039c30 con 0x7f05ac0d2970 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.048+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05ac1595f0 con 0x7f05ac0d2970 2024-03-23T14:36:11.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.049+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05ac0cae80 con 0x7f05ac0d2970 2024-03-23T14:36:11.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.050+0000 7f05a27fc640 1 -- 172.21.15.161:0/2475339887 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f05a804b400 con 0x7f05ac0d2970 2024-03-23T14:36:11.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.050+0000 7f05a27fc640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f059006b810 0x7f059006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.050+0000 7f05a27fc640 1 -- 172.21.15.161:0/2475339887 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f05a80ac400 con 0x7f05ac0d2970 2024-03-23T14:36:11.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.050+0000 7f05b1eda640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f059006b810 0x7f059006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.050+0000 7f05b1eda640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f059006b810 0x7f059006dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f059c005880 tx=0x7f059c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.054+0000 7f05a27fc640 1 -- 172.21.15.161:0/2475339887 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f05a8076ed0 con 0x7f05ac0d2970 2024-03-23T14:36:11.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.224+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f05ac0c9f70 con 0x7f059006b810 2024-03-23T14:36:11.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.230+0000 7f05a27fc640 1 -- 172.21.15.161:0/2475339887 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239826 (secure 0 0 0) 0x7f05ac0c9f70 con 0x7f059006b810 2024-03-23T14:36:11.228 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:11.229 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:36:11.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f059006b810 msgr2=0x7f059006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f059006b810 0x7f059006dcd0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f059c005880 tx=0x7f059c009040 comp rx=0 tx=0).stop 2024-03-23T14:36:11.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05ac0d2970 msgr2=0x7f05ac0b47f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05ac0d2970 0x7f05ac0b47f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f05a8000c00 tx=0x7f05a80046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 shutdown_connections 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f059006b810 0x7f059006dcd0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05ac15e400 0x7f05ac0c8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05ac0d2970 0x7f05ac0b47f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 --2- 172.21.15.161:0/2475339887 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05ac0d1470 0x7f05ac0b42b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 >> 172.21.15.161:0/2475339887 conn(0x7f05ac09d190 msgr2=0x7f05ac15d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 shutdown_connections 2024-03-23T14:36:11.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.232+0000 7f05b413e640 1 -- 172.21.15.161:0/2475339887 wait complete. 2024-03-23T14:36:11.239 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-03-23T14:36:10.484585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219920,"kb_used_data":3748,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754800,"statfs":{"total":773094113280,"available":772868915200,"internally_reserved":0,"allocated":3837952,"data_stored":2181477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.568230"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921146+0000","last_change":"2024-03-23T14:36:04.921146+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921146+0000","last_undegraded":"2024-03-23T14:36:04.921146+0000","last_fullsized":"2024-03-23T14:36:04.921146+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919587+0000","last_change":"2024-03-23T14:36:04.919587+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919587+0000","last_undegraded":"2024-03-23T14:36:04.919587+0000","last_fullsized":"2024-03-23T14:36:04.919587+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922002+0000","last_change":"2024-03-23T14:36:04.922002+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922002+0000","last_undegraded":"2024-03-23T14:36:04.922002+0000","last_fullsized":"2024-03-23T14:36:04.922002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920050+0000","last_change":"2024-03-23T14:36:04.920050+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920050+0000","last_undegraded":"2024-03-23T14:36:04.920050+0000","last_fullsized":"2024-03-23T14:36:04.920050+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919319+0000","last_change":"2024-03-23T14:36:04.919319+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919319+0000","last_undegraded":"2024-03-23T14:36:04.919319+0000","last_fullsized":"2024-03-23T14:36:04.919319+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918554+0000","last_change":"2024-03-23T14:36:04.918554+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918554+0000","last_undegraded":"2024-03-23T14:36:04.918554+0000","last_fullsized":"2024-03-23T14:36:04.918554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921218+0000","last_change":"2024-03-23T14:36:04.921218+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921218+0000","last_undegraded":"2024-03-23T14:36:04.921218+0000","last_fullsized":"2024-03-23T14:36:04.921218+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919010+0000","last_change":"2024-03-23T14:36:04.919010+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919010+0000","last_undegraded":"2024-03-23T14:36:04.919010+0000","last_fullsized":"2024-03-23T14:36:04.919010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920263+0000","last_change":"2024-03-23T14:36:04.920263+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920263+0000","last_undegraded":"2024-03-23T14:36:04.920263+0000","last_fullsized":"2024-03-23T14:36:04.920263+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919149+0000","last_change":"2024-03-23T14:36:04.919149+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919149+0000","last_undegraded":"2024-03-23T14:36:04.919149+0000","last_fullsized":"2024-03-23T14:36:04.919149+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921107+0000","last_change":"2024-03-23T14:36:04.921107+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921107+0000","last_undegraded":"2024-03-23T14:36:04.921107+0000","last_fullsized":"2024-03-23T14:36:04.921107+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920278+0000","last_change":"2024-03-23T14:36:04.920278+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920278+0000","last_undegraded":"2024-03-23T14:36:04.920278+0000","last_fullsized":"2024-03-23T14:36:04.920278+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918667+0000","last_change":"2024-03-23T14:36:04.918667+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918667+0000","last_undegraded":"2024-03-23T14:36:04.918667+0000","last_fullsized":"2024-03-23T14:36:04.918667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918094+0000","last_change":"2024-03-23T14:36:04.918094+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918094+0000","last_undegraded":"2024-03-23T14:36:04.918094+0000","last_fullsized":"2024-03-23T14:36:04.918094+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921589+0000","last_change":"2024-03-23T14:36:04.921589+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921589+0000","last_undegraded":"2024-03-23T14:36:04.921589+0000","last_fullsized":"2024-03-23T14:36:04.921589+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920831+0000","last_change":"2024-03-23T14:36:04.920831+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920831+0000","last_undegraded":"2024-03-23T14:36:04.920831+0000","last_fullsized":"2024-03-23T14:36:04.920831+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918060+0000","last_change":"2024-03-23T14:36:04.918060+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918060+0000","last_undegraded":"2024-03-23T14:36:04.918060+0000","last_fullsized":"2024-03-23T14:36:04.918060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919674+0000","last_change":"2024-03-23T14:36:04.919674+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919674+0000","last_undegraded":"2024-03-23T14:36:04.919674+0000","last_fullsized":"2024-03-23T14:36:04.919674+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919612+0000","last_change":"2024-03-23T14:36:04.919612+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919612+0000","last_undegraded":"2024-03-23T14:36:04.919612+0000","last_fullsized":"2024-03-23T14:36:04.919612+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918594+0000","last_change":"2024-03-23T14:36:04.918594+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918594+0000","last_undegraded":"2024-03-23T14:36:04.918594+0000","last_fullsized":"2024-03-23T14:36:04.918594+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918956+0000","last_change":"2024-03-23T14:36:04.918956+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918956+0000","last_undegraded":"2024-03-23T14:36:04.918956+0000","last_fullsized":"2024-03-23T14:36:04.918956+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921160+0000","last_change":"2024-03-23T14:36:04.921160+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921160+0000","last_undegraded":"2024-03-23T14:36:04.921160+0000","last_fullsized":"2024-03-23T14:36:04.921160+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920128+0000","last_change":"2024-03-23T14:36:04.920128+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920128+0000","last_undegraded":"2024-03-23T14:36:04.920128+0000","last_fullsized":"2024-03-23T14:36:04.920128+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922010+0000","last_change":"2024-03-23T14:36:04.922010+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922010+0000","last_undegraded":"2024-03-23T14:36:04.922010+0000","last_fullsized":"2024-03-23T14:36:04.922010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917607+0000","last_change":"2024-03-23T14:36:04.917607+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917607+0000","last_undegraded":"2024-03-23T14:36:04.917607+0000","last_fullsized":"2024-03-23T14:36:04.917607+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918696+0000","last_change":"2024-03-23T14:36:04.918696+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918696+0000","last_undegraded":"2024-03-23T14:36:04.918696+0000","last_fullsized":"2024-03-23T14:36:04.918696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917667+0000","last_change":"2024-03-23T14:36:04.917667+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917667+0000","last_undegraded":"2024-03-23T14:36:04.917667+0000","last_fullsized":"2024-03-23T14:36:04.917667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918505+0000","last_change":"2024-03-23T14:36:04.918505+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918505+0000","last_undegraded":"2024-03-23T14:36:04.918505+0000","last_fullsized":"2024-03-23T14:36:04.918505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917659+0000","last_change":"2024-03-23T14:36:04.917659+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917659+0000","last_undegraded":"2024-03-23T14:36:04.917659+0000","last_fullsized":"2024-03-23T14:36:04.917659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918561+0000","last_change":"2024-03-23T14:36:04.918561+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918561+0000","last_undegraded":"2024-03-23T14:36:04.918561+0000","last_fullsized":"2024-03-23T14:36:04.918561+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922105+0000","last_change":"2024-03-23T14:36:04.922105+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922105+0000","last_undegraded":"2024-03-23T14:36:04.922105+0000","last_fullsized":"2024-03-23T14:36:04.922105+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921452+0000","last_change":"2024-03-23T14:36:04.921452+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921452+0000","last_undegraded":"2024-03-23T14:36:04.921452+0000","last_fullsized":"2024-03-23T14:36:04.921452+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-23T14:36:04.922822+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:36:04.922822+0000","last_peered":"2024-03-23T14:36:04.922822+0000","last_clean":"2024-03-23T14:36:04.922822+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:36:04.922822+0000","last_undegraded":"2024-03-23T14:36:04.922822+0000","last_fullsized":"2024-03-23T14:36:04.922822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":917504,"data_stored":711825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":121457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":121457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:36:11.243 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:36:11.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.384+0000 7f3cbbd3e640 1 Processor -- start 2024-03-23T14:36:11.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.385+0000 7f3cbbd3e640 1 -- start start 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.385+0000 7f3cbbd3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 0x7f3cb40cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.385+0000 7f3cbbd3e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3cb40cf9d0 0x7f3cb40d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.385+0000 7f3cbbd3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3cb40d0460 con 0x7f3cb40d1470 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.385+0000 7f3cbbd3e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3cb40d1050 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.385+0000 7f3cbbd3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb40d06b0 con 0x7f3cb415e400 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb92d9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3cb40cf9d0 0x7f3cb40d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb92d9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3cb40cf9d0 0x7f3cb40d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36456/0 (socket says 172.21.15.161:36456) 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb92d9640 1 -- 172.21.15.161:0/996823481 learned_addr learned my addr 172.21.15.161:0/996823481 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb9ada640 1 --2- 172.21.15.161:0/996823481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 0x7f3cb40cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb92d9640 1 -- 172.21.15.161:0/996823481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 msgr2=0x7f3cb40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:11.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb92d9640 1 --2- 172.21.15.161:0/996823481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 0x7f3cb40cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb92d9640 1 -- 172.21.15.161:0/996823481 >> v1:172.21.15.183:6789/0 conn(0x7f3cb40d1470 legacy=0x7f3cb40b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb8ad8640 1 -- 172.21.15.161:0/996823481 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3035226231 0 0) 0x7f3cb40d0460 con 0x7f3cb40d1470 2024-03-23T14:36:11.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.386+0000 7f3cb92d9640 1 -- 172.21.15.161:0/996823481 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb40d0990 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cb92d9640 1 --2- 172.21.15.161:0/996823481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb40d2430 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009870 tx=0x7f3cb002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fdcb27ae52253f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cb8ad8640 1 -- 172.21.15.161:0/996823481 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0030840 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cb8ad8640 1 -- 172.21.15.161:0/996823481 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3cb00309a0 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cb8ad8640 1 -- 172.21.15.161:0/996823481 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cb0030d30 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/996823481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 msgr2=0x7f3cb40d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/996823481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb40d2430 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0009870 tx=0x7f3cb002b980 comp rx=0 tx=0).stop 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/996823481 shutdown_connections 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/996823481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb40d2430 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/996823481 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 0x7f3cb40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/996823481 >> 172.21.15.161:0/996823481 conn(0x7f3cb409d190 msgr2=0x7f3cb40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.387+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/996823481 shutdown_connections 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.388+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/996823481 wait complete. 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cbbd3e640 1 Processor -- start 2024-03-23T14:36:11.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cbbd3e640 1 -- start start 2024-03-23T14:36:11.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cbbd3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb41f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cbbd3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cb40d1470 0x7f3cb4160d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cbbd3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 0x7f3cb41612d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cbbd3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb40cb030 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cbbd3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb40cb310 con 0x7f3cb40d1470 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cbbd3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3cb40cb5f0 con 0x7f3cb415e400 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cb9ada640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb41f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.389+0000 7f3cb9ada640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb41f3420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36466/0 (socket says 172.21.15.161:36466) 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb9ada640 1 -- 172.21.15.161:0/474424332 learned_addr learned my addr 172.21.15.161:0/474424332 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb92d9640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cb40d1470 0x7f3cb4160d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cba2db640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 0x7f3cb41612d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb9ada640 1 -- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 msgr2=0x7f3cb41612d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb9ada640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 0x7f3cb41612d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb9ada640 1 -- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cb40d1470 msgr2=0x7f3cb4160d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb9ada640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cb40d1470 0x7f3cb4160d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb9ada640 1 -- 172.21.15.161:0/474424332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb00094d0 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb92d9640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cb40d1470 0x7f3cb4160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:11.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cb9ada640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb41f3420 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400e870 tx=0x7f3ca400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3caa7fc640 1 -- 172.21.15.161:0/474424332 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca40094f0 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3caa7fc640 1 -- 172.21.15.161:0/474424332 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ca400c940 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb40cb8d0 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3caa7fc640 1 -- 172.21.15.161:0/474424332 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ca400ccf0 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.390+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb41f3960 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.391+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb40a1e00 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.392+0000 7f3caa7fc640 1 -- 172.21.15.161:0/474424332 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3ca4013470 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.392+0000 7f3caa7fc640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3c9806b840 0x7f3c9806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.392+0000 7f3caa7fc640 1 -- 172.21.15.161:0/474424332 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f3ca408a140 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.392+0000 7f3cb92d9640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3c9806b840 0x7f3c9806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.393+0000 7f3cb92d9640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3c9806b840 0x7f3c9806dd00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0036000 tx=0x7f3cb0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.397+0000 7f3caa7fc640 1 -- 172.21.15.161:0/474424332 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3ca4054ca0 con 0x7f3cb40cf9d0 2024-03-23T14:36:11.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.566+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3cb40cb480 con 0x7f3c9806b840 2024-03-23T14:36:11.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.571+0000 7f3caa7fc640 1 -- 172.21.15.161:0/474424332 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239826 (secure 0 0 0) 0x7f3cb40cb480 con 0x7f3c9806b840 2024-03-23T14:36:11.569 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:11.570 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:36:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3c9806b840 msgr2=0x7f3c9806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3c9806b840 0x7f3c9806dd00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f3cb0036000 tx=0x7f3cb0008030 comp rx=0 tx=0).stop 2024-03-23T14:36:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 msgr2=0x7f3cb41f3420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb41f3420 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3ca400e870 tx=0x7f3ca400ed30 comp rx=0 tx=0).stop 2024-03-23T14:36:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 shutdown_connections 2024-03-23T14:36:11.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3c9806b840 0x7f3c9806dd00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3cb415e400 0x7f3cb41612d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3cb40d1470 0x7f3cb4160d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 --2- 172.21.15.161:0/474424332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3cb40cf9d0 0x7f3cb41f3420 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.574+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 >> 172.21.15.161:0/474424332 conn(0x7f3cb409d190 msgr2=0x7f3cb406b9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:11.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.575+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 shutdown_connections 2024-03-23T14:36:11.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.575+0000 7f3cbbd3e640 1 -- 172.21.15.161:0/474424332 wait complete. 2024-03-23T14:36:11.581 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-03-23T14:36:10.484585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219920,"kb_used_data":3748,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754800,"statfs":{"total":773094113280,"available":772868915200,"internally_reserved":0,"allocated":3837952,"data_stored":2181477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.568230"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921146+0000","last_change":"2024-03-23T14:36:04.921146+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921146+0000","last_undegraded":"2024-03-23T14:36:04.921146+0000","last_fullsized":"2024-03-23T14:36:04.921146+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919587+0000","last_change":"2024-03-23T14:36:04.919587+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919587+0000","last_undegraded":"2024-03-23T14:36:04.919587+0000","last_fullsized":"2024-03-23T14:36:04.919587+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922002+0000","last_change":"2024-03-23T14:36:04.922002+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922002+0000","last_undegraded":"2024-03-23T14:36:04.922002+0000","last_fullsized":"2024-03-23T14:36:04.922002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920050+0000","last_change":"2024-03-23T14:36:04.920050+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920050+0000","last_undegraded":"2024-03-23T14:36:04.920050+0000","last_fullsized":"2024-03-23T14:36:04.920050+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919319+0000","last_change":"2024-03-23T14:36:04.919319+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919319+0000","last_undegraded":"2024-03-23T14:36:04.919319+0000","last_fullsized":"2024-03-23T14:36:04.919319+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918554+0000","last_change":"2024-03-23T14:36:04.918554+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918554+0000","last_undegraded":"2024-03-23T14:36:04.918554+0000","last_fullsized":"2024-03-23T14:36:04.918554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921218+0000","last_change":"2024-03-23T14:36:04.921218+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921218+0000","last_undegraded":"2024-03-23T14:36:04.921218+0000","last_fullsized":"2024-03-23T14:36:04.921218+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919010+0000","last_change":"2024-03-23T14:36:04.919010+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919010+0000","last_undegraded":"2024-03-23T14:36:04.919010+0000","last_fullsized":"2024-03-23T14:36:04.919010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920263+0000","last_change":"2024-03-23T14:36:04.920263+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920263+0000","last_undegraded":"2024-03-23T14:36:04.920263+0000","last_fullsized":"2024-03-23T14:36:04.920263+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919149+0000","last_change":"2024-03-23T14:36:04.919149+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919149+0000","last_undegraded":"2024-03-23T14:36:04.919149+0000","last_fullsized":"2024-03-23T14:36:04.919149+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921107+0000","last_change":"2024-03-23T14:36:04.921107+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921107+0000","last_undegraded":"2024-03-23T14:36:04.921107+0000","last_fullsized":"2024-03-23T14:36:04.921107+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920278+0000","last_change":"2024-03-23T14:36:04.920278+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920278+0000","last_undegraded":"2024-03-23T14:36:04.920278+0000","last_fullsized":"2024-03-23T14:36:04.920278+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918667+0000","last_change":"2024-03-23T14:36:04.918667+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918667+0000","last_undegraded":"2024-03-23T14:36:04.918667+0000","last_fullsized":"2024-03-23T14:36:04.918667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918094+0000","last_change":"2024-03-23T14:36:04.918094+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918094+0000","last_undegraded":"2024-03-23T14:36:04.918094+0000","last_fullsized":"2024-03-23T14:36:04.918094+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921589+0000","last_change":"2024-03-23T14:36:04.921589+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921589+0000","last_undegraded":"2024-03-23T14:36:04.921589+0000","last_fullsized":"2024-03-23T14:36:04.921589+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920831+0000","last_change":"2024-03-23T14:36:04.920831+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920831+0000","last_undegraded":"2024-03-23T14:36:04.920831+0000","last_fullsized":"2024-03-23T14:36:04.920831+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918060+0000","last_change":"2024-03-23T14:36:04.918060+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918060+0000","last_undegraded":"2024-03-23T14:36:04.918060+0000","last_fullsized":"2024-03-23T14:36:04.918060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919674+0000","last_change":"2024-03-23T14:36:04.919674+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919674+0000","last_undegraded":"2024-03-23T14:36:04.919674+0000","last_fullsized":"2024-03-23T14:36:04.919674+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919612+0000","last_change":"2024-03-23T14:36:04.919612+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919612+0000","last_undegraded":"2024-03-23T14:36:04.919612+0000","last_fullsized":"2024-03-23T14:36:04.919612+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918594+0000","last_change":"2024-03-23T14:36:04.918594+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918594+0000","last_undegraded":"2024-03-23T14:36:04.918594+0000","last_fullsized":"2024-03-23T14:36:04.918594+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918956+0000","last_change":"2024-03-23T14:36:04.918956+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918956+0000","last_undegraded":"2024-03-23T14:36:04.918956+0000","last_fullsized":"2024-03-23T14:36:04.918956+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921160+0000","last_change":"2024-03-23T14:36:04.921160+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921160+0000","last_undegraded":"2024-03-23T14:36:04.921160+0000","last_fullsized":"2024-03-23T14:36:04.921160+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920128+0000","last_change":"2024-03-23T14:36:04.920128+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920128+0000","last_undegraded":"2024-03-23T14:36:04.920128+0000","last_fullsized":"2024-03-23T14:36:04.920128+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922010+0000","last_change":"2024-03-23T14:36:04.922010+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922010+0000","last_undegraded":"2024-03-23T14:36:04.922010+0000","last_fullsized":"2024-03-23T14:36:04.922010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917607+0000","last_change":"2024-03-23T14:36:04.917607+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917607+0000","last_undegraded":"2024-03-23T14:36:04.917607+0000","last_fullsized":"2024-03-23T14:36:04.917607+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918696+0000","last_change":"2024-03-23T14:36:04.918696+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918696+0000","last_undegraded":"2024-03-23T14:36:04.918696+0000","last_fullsized":"2024-03-23T14:36:04.918696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917667+0000","last_change":"2024-03-23T14:36:04.917667+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917667+0000","last_undegraded":"2024-03-23T14:36:04.917667+0000","last_fullsized":"2024-03-23T14:36:04.917667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918505+0000","last_change":"2024-03-23T14:36:04.918505+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918505+0000","last_undegraded":"2024-03-23T14:36:04.918505+0000","last_fullsized":"2024-03-23T14:36:04.918505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917659+0000","last_change":"2024-03-23T14:36:04.917659+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917659+0000","last_undegraded":"2024-03-23T14:36:04.917659+0000","last_fullsized":"2024-03-23T14:36:04.917659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918561+0000","last_change":"2024-03-23T14:36:04.918561+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918561+0000","last_undegraded":"2024-03-23T14:36:04.918561+0000","last_fullsized":"2024-03-23T14:36:04.918561+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922105+0000","last_change":"2024-03-23T14:36:04.922105+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922105+0000","last_undegraded":"2024-03-23T14:36:04.922105+0000","last_fullsized":"2024-03-23T14:36:04.922105+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921452+0000","last_change":"2024-03-23T14:36:04.921452+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921452+0000","last_undegraded":"2024-03-23T14:36:04.921452+0000","last_fullsized":"2024-03-23T14:36:04.921452+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-23T14:36:04.922822+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:36:04.922822+0000","last_peered":"2024-03-23T14:36:04.922822+0000","last_clean":"2024-03-23T14:36:04.922822+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:36:04.922822+0000","last_undegraded":"2024-03-23T14:36:04.922822+0000","last_fullsized":"2024-03-23T14:36:04.922822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":917504,"data_stored":711825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":121457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":121457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:36:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3f is not active+clean 2024-03-23T14:36:11.585 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3e is not active+clean 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3d is not active+clean 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3c is not active+clean 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3b is not active+clean 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3a is not active+clean 2024-03-23T14:36:11.586 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.39 is not active+clean 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.39', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.38 is not active+clean 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.38', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.37 is not active+clean 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.37', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.36 is not active+clean 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.36', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.35 is not active+clean 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.35', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.587 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.34 is not active+clean 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.34', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.33 is not active+clean 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.33', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.32 is not active+clean 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.32', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.31 is not active+clean 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.31', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.30 is not active+clean 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.30', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2f is not active+clean 2024-03-23T14:36:11.588 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2e is not active+clean 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2d is not active+clean 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2c is not active+clean 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2b is not active+clean 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2a is not active+clean 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.589 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.29 is not active+clean 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.29', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.28 is not active+clean 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.28', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.27 is not active+clean 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.27', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.26 is not active+clean 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.26', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.25 is not active+clean 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.25', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.590 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.24 is not active+clean 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.24', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.23 is not active+clean 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.23', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.22 is not active+clean 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.22', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.21 is not active+clean 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.21', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.20 is not active+clean 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.20', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1e is not active+clean 2024-03-23T14:36:11.591 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1f is not active+clean 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.921146+0000', 'last_change': '2024-03-23T14:36:04.921146+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.921146+0000', 'last_undegraded': '2024-03-23T14:36:04.921146+0000', 'last_fullsized': '2024-03-23T14:36:04.921146+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 5], 'acting': [6, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1f is not active+clean 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1e is not active+clean 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.919587+0000', 'last_change': '2024-03-23T14:36:04.919587+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.919587+0000', 'last_undegraded': '2024-03-23T14:36:04.919587+0000', 'last_fullsized': '2024-03-23T14:36:04.919587+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 7], 'acting': [0, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1c is not active+clean 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1d is not active+clean 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.922002+0000', 'last_change': '2024-03-23T14:36:04.922002+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.922002+0000', 'last_undegraded': '2024-03-23T14:36:04.922002+0000', 'last_fullsized': '2024-03-23T14:36:04.922002+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T14:36:11.592 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1d is not active+clean 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1c is not active+clean 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.920050+0000', 'last_change': '2024-03-23T14:36:04.920050+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.920050+0000', 'last_undegraded': '2024-03-23T14:36:04.920050+0000', 'last_fullsized': '2024-03-23T14:36:04.920050+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [2, 1], 'acting': [2, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 2, 'acting_primary': 2, 'purged_snaps': []} 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1a is not active+clean 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1b is not active+clean 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.919319+0000', 'last_change': '2024-03-23T14:36:04.919319+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.919319+0000', 'last_undegraded': '2024-03-23T14:36:04.919319+0000', 'last_fullsized': '2024-03-23T14:36:04.919319+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1b is not active+clean 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1a is not active+clean 2024-03-23T14:36:11.593 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918554+0000', 'last_change': '2024-03-23T14:36:04.918554+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918554+0000', 'last_undegraded': '2024-03-23T14:36:04.918554+0000', 'last_fullsized': '2024-03-23T14:36:04.918554+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.18 is not active+clean 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.18', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.19 is not active+clean 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.19', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.921218+0000', 'last_change': '2024-03-23T14:36:04.921218+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.921218+0000', 'last_undegraded': '2024-03-23T14:36:04.921218+0000', 'last_fullsized': '2024-03-23T14:36:04.921218+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 2], 'acting': [3, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.19 is not active+clean 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.19', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.18 is not active+clean 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.18', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.919010+0000', 'last_change': '2024-03-23T14:36:04.919010+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.919010+0000', 'last_undegraded': '2024-03-23T14:36:04.919010+0000', 'last_fullsized': '2024-03-23T14:36:04.919010+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.16 is not active+clean 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.16', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.594 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.17 is not active+clean 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.17', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.920263+0000', 'last_change': '2024-03-23T14:36:04.920263+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.920263+0000', 'last_undegraded': '2024-03-23T14:36:04.920263+0000', 'last_fullsized': '2024-03-23T14:36:04.920263+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.17 is not active+clean 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.17', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.16 is not active+clean 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.16', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.919149+0000', 'last_change': '2024-03-23T14:36:04.919149+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.919149+0000', 'last_undegraded': '2024-03-23T14:36:04.919149+0000', 'last_fullsized': '2024-03-23T14:36:04.919149+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 3], 'acting': [0, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.14 is not active+clean 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.14', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.15 is not active+clean 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.15', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.921107+0000', 'last_change': '2024-03-23T14:36:04.921107+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.921107+0000', 'last_undegraded': '2024-03-23T14:36:04.921107+0000', 'last_fullsized': '2024-03-23T14:36:04.921107+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.15 is not active+clean 2024-03-23T14:36:11.595 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.15', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.14 is not active+clean 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.14', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.920278+0000', 'last_change': '2024-03-23T14:36:04.920278+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.920278+0000', 'last_undegraded': '2024-03-23T14:36:04.920278+0000', 'last_fullsized': '2024-03-23T14:36:04.920278+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.12 is not active+clean 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.12', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.13 is not active+clean 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.13', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918667+0000', 'last_change': '2024-03-23T14:36:04.918667+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918667+0000', 'last_undegraded': '2024-03-23T14:36:04.918667+0000', 'last_fullsized': '2024-03-23T14:36:04.918667+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.13 is not active+clean 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.13', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.12 is not active+clean 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.12', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918094+0000', 'last_change': '2024-03-23T14:36:04.918094+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918094+0000', 'last_undegraded': '2024-03-23T14:36:04.918094+0000', 'last_fullsized': '2024-03-23T14:36:04.918094+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 6], 'acting': [1, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.10 is not active+clean 2024-03-23T14:36:11.596 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.10', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.11 is not active+clean 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.11', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.921589+0000', 'last_change': '2024-03-23T14:36:04.921589+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.921589+0000', 'last_undegraded': '2024-03-23T14:36:04.921589+0000', 'last_fullsized': '2024-03-23T14:36:04.921589+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.11 is not active+clean 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.11', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.10 is not active+clean 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.10', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.920831+0000', 'last_change': '2024-03-23T14:36:04.920831+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.920831+0000', 'last_undegraded': '2024-03-23T14:36:04.920831+0000', 'last_fullsized': '2024-03-23T14:36:04.920831+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 1], 'acting': [3, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.e is not active+clean 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.e', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.f is not active+clean 2024-03-23T14:36:11.597 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.f', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918060+0000', 'last_change': '2024-03-23T14:36:04.918060+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918060+0000', 'last_undegraded': '2024-03-23T14:36:04.918060+0000', 'last_fullsized': '2024-03-23T14:36:04.918060+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 3], 'acting': [1, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.f is not active+clean 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.f', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.e is not active+clean 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.e', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.919674+0000', 'last_change': '2024-03-23T14:36:04.919674+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.919674+0000', 'last_undegraded': '2024-03-23T14:36:04.919674+0000', 'last_fullsized': '2024-03-23T14:36:04.919674+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 6], 'acting': [4, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.c is not active+clean 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.c', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.d is not active+clean 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.d', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.919612+0000', 'last_change': '2024-03-23T14:36:04.919612+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.919612+0000', 'last_undegraded': '2024-03-23T14:36:04.919612+0000', 'last_fullsized': '2024-03-23T14:36:04.919612+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 2], 'acting': [4, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.d is not active+clean 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.d', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.598 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.c is not active+clean 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.c', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918594+0000', 'last_change': '2024-03-23T14:36:04.918594+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918594+0000', 'last_undegraded': '2024-03-23T14:36:04.918594+0000', 'last_fullsized': '2024-03-23T14:36:04.918594+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 3], 'acting': [4, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.a is not active+clean 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.a', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.b is not active+clean 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.b', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918956+0000', 'last_change': '2024-03-23T14:36:04.918956+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918956+0000', 'last_undegraded': '2024-03-23T14:36:04.918956+0000', 'last_fullsized': '2024-03-23T14:36:04.918956+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.b is not active+clean 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.b', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.a is not active+clean 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.a', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.921160+0000', 'last_change': '2024-03-23T14:36:04.921160+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.921160+0000', 'last_undegraded': '2024-03-23T14:36:04.921160+0000', 'last_fullsized': '2024-03-23T14:36:04.921160+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 1], 'acting': [6, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.8 is not active+clean 2024-03-23T14:36:11.599 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.8', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.9 is not active+clean 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.9', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.920128+0000', 'last_change': '2024-03-23T14:36:04.920128+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.920128+0000', 'last_undegraded': '2024-03-23T14:36:04.920128+0000', 'last_fullsized': '2024-03-23T14:36:04.920128+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 1], 'acting': [4, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.9 is not active+clean 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.9', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.8 is not active+clean 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.8', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.922010+0000', 'last_change': '2024-03-23T14:36:04.922010+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.922010+0000', 'last_undegraded': '2024-03-23T14:36:04.922010+0000', 'last_fullsized': '2024-03-23T14:36:04.922010+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [5, 7], 'acting': [5, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 5, 'acting_primary': 5, 'purged_snaps': []} 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.6 is not active+clean 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.6', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.7 is not active+clean 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.7', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.917607+0000', 'last_change': '2024-03-23T14:36:04.917607+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.917607+0000', 'last_undegraded': '2024-03-23T14:36:04.917607+0000', 'last_fullsized': '2024-03-23T14:36:04.917607+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-23T14:36:11.600 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.7 is not active+clean 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.7', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.6 is not active+clean 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.6', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918696+0000', 'last_change': '2024-03-23T14:36:04.918696+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918696+0000', 'last_undegraded': '2024-03-23T14:36:04.918696+0000', 'last_fullsized': '2024-03-23T14:36:04.918696+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 1], 'acting': [0, 1], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.5 is not active+clean 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.5', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.4 is not active+clean 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.4', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.917667+0000', 'last_change': '2024-03-23T14:36:04.917667+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.917667+0000', 'last_undegraded': '2024-03-23T14:36:04.917667+0000', 'last_fullsized': '2024-03-23T14:36:04.917667+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 2], 'acting': [1, 2], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.2 is not active+clean 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.2', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.3 is not active+clean 2024-03-23T14:36:11.601 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.3', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918505+0000', 'last_change': '2024-03-23T14:36:04.918505+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918505+0000', 'last_undegraded': '2024-03-23T14:36:04.918505+0000', 'last_fullsized': '2024-03-23T14:36:04.918505+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [0, 5], 'acting': [0, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 0, 'acting_primary': 0, 'purged_snaps': []} 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.3 is not active+clean 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.3', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.2 is not active+clean 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.2', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.917659+0000', 'last_change': '2024-03-23T14:36:04.917659+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.917659+0000', 'last_undegraded': '2024-03-23T14:36:04.917659+0000', 'last_fullsized': '2024-03-23T14:36:04.917659+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [1, 5], 'acting': [1, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 1, 'acting_primary': 1, 'purged_snaps': []} 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.0 is not active+clean 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.0', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.1 is not active+clean 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.1', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.918561+0000', 'last_change': '2024-03-23T14:36:04.918561+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.918561+0000', 'last_undegraded': '2024-03-23T14:36:04.918561+0000', 'last_fullsized': '2024-03-23T14:36:04.918561+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [4, 5], 'acting': [4, 5], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 4, 'acting_primary': 4, 'purged_snaps': []} 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.1 is not active+clean 2024-03-23T14:36:11.602 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.1', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.0 is not active+clean 2024-03-23T14:36:11.603 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.0', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.922105+0000', 'last_change': '2024-03-23T14:36:04.922105+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.922105+0000', 'last_undegraded': '2024-03-23T14:36:04.922105+0000', 'last_fullsized': '2024-03-23T14:36:04.922105+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [3, 7], 'acting': [3, 7], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 3, 'acting_primary': 3, 'purged_snaps': []} 2024-03-23T14:36:11.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 5.4 is not active+clean 2024-03-23T14:36:11.603 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '5.4', 'version': "0'0", 'reported_seq': 0, 'reported_epoch': 0, 'state': 'unknown', 'last_fresh': '2024-03-23T14:36:05.914757+0000', 'last_change': '2024-03-23T14:36:05.914757+0000', 'last_active': '2024-03-23T14:36:05.914757+0000', 'last_peered': '2024-03-23T14:36:05.914757+0000', 'last_clean': '2024-03-23T14:36:05.914757+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:05.914757+0000', 'last_undegraded': '2024-03-23T14:36:05.914757+0000', 'last_fullsized': '2024-03-23T14:36:05.914757+0000', 'mapping_epoch': 0, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 0, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:05.914757+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': '--', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [], 'acting': [], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': -1, 'acting_primary': -1, 'purged_snaps': []} 2024-03-23T14:36:11.603 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.5 is not active+clean 2024-03-23T14:36:11.603 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.5', 'version': "0'0", 'reported_seq': 1, 'reported_epoch': 30, 'state': 'creating+peering', 'last_fresh': '2024-03-23T14:36:04.921452+0000', 'last_change': '2024-03-23T14:36:04.921452+0000', 'last_active': '2024-03-23T14:36:04.910778+0000', 'last_peered': '2024-03-23T14:36:04.910778+0000', 'last_clean': '2024-03-23T14:36:04.910778+0000', 'last_became_active': '0.000000', 'last_became_peered': '0.000000', 'last_unstale': '2024-03-23T14:36:04.921452+0000', 'last_undegraded': '2024-03-23T14:36:04.921452+0000', 'last_fullsized': '2024-03-23T14:36:04.921452+0000', 'mapping_epoch': 30, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 30, 'last_epoch_clean': 0, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'last_clean_scrub_stamp': '2024-03-23T14:36:04.910778+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [6, 4], 'acting': [6, 4], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 6, 'acting_primary': 6, 'purged_snaps': []} 2024-03-23T14:36:11.603 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-03-23T14:36:11.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.727+0000 7f2ae5a2a640 1 Processor -- start 2024-03-23T14:36:11.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.727+0000 7f2ae5a2a640 1 -- start start 2024-03-23T14:36:11.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2ae5a2a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ae015e400 0x7f2ae00cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2ae5a2a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2ae00cf9d0 0x7f2ae00d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2ae5a2a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ae00d0460 con 0x7f2ae00d1470 2024-03-23T14:36:11.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2ae5a2a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2ae00d1050 con 0x7f2ae015e400 2024-03-23T14:36:11.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2ae5a2a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f2ae00d06b0 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2adeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ae015e400 0x7f2ae00cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2adeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2ae015e400 0x7f2ae00cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36472/0 (socket says 172.21.15.161:36472) 2024-03-23T14:36:11.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2adeffd640 1 -- 172.21.15.161:0/2090986681 learned_addr learned my addr 172.21.15.161:0/2090986681 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:11.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2ad7fff640 1 --2- 172.21.15.161:0/2090986681 >> v2:172.21.15.183:3300/0 conn(0x7f2ae00cf9d0 0x7f2ae00d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2adeffd640 1 -- 172.21.15.161:0/2090986681 >> v2:172.21.15.183:3300/0 conn(0x7f2ae00cf9d0 msgr2=0x7f2ae00d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:11.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2adeffd640 1 --2- 172.21.15.161:0/2090986681 >> v2:172.21.15.183:3300/0 conn(0x7f2ae00cf9d0 0x7f2ae00d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2adeffd640 1 -- 172.21.15.161:0/2090986681 >> v1:172.21.15.183:6789/0 conn(0x7f2ae00d1470 legacy=0x7f2ae00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.728+0000 7f2adeffd640 1 -- 172.21.15.161:0/2090986681 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ae00d0990 con 0x7f2ae015e400 2024-03-23T14:36:11.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2adeffd640 1 --2- 172.21.15.161:0/2090986681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae015e400 0x7f2ae00cf490 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00b6a20 tx=0x7f2ad002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=374998c32472ef06 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ade7fc640 1 -- 172.21.15.161:0/2090986681 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0030860 con 0x7f2ae015e400 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ade7fc640 1 -- 172.21.15.161:0/2090986681 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad00309c0 con 0x7f2ae015e400 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ade7fc640 1 -- 172.21.15.161:0/2090986681 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0030d50 con 0x7f2ae015e400 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/2090986681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae015e400 msgr2=0x7f2ae00cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/2090986681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae015e400 0x7f2ae00cf490 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00b6a20 tx=0x7f2ad002b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/2090986681 shutdown_connections 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/2090986681 >> v2:172.21.15.183:3300/0 conn(0x7f2ae00cf9d0 0x7f2ae00d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/2090986681 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae015e400 0x7f2ae00cf490 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.729+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/2090986681 >> 172.21.15.161:0/2090986681 conn(0x7f2ae009d190 msgr2=0x7f2ae00d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.730+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/2090986681 shutdown_connections 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.730+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/2090986681 wait complete. 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2ae5a2a640 1 Processor -- start 2024-03-23T14:36:11.727 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2ae5a2a640 1 -- start start 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2ae5a2a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00cf9d0 0x7f2ae00cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2ae5a2a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ae00d1470 0x7f2ae00c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2ae5a2a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ae015e400 0x7f2ae00c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2ae5a2a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae00ca340 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2ae5a2a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2ae00cbff0 con 0x7f2ae00d1470 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2ae5a2a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2ae00c9040 con 0x7f2ae015e400 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2adeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00cf9d0 0x7f2ae00cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2adeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00cf9d0 0x7f2ae00cb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36484/0 (socket says 172.21.15.161:36484) 2024-03-23T14:36:11.730 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.731+0000 7f2adeffd640 1 -- 172.21.15.161:0/3153468781 learned_addr learned my addr 172.21.15.161:0/3153468781 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2ad7fff640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ae00d1470 0x7f2ae00c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adf7fe640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ae015e400 0x7f2ae00c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adeffd640 1 -- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ae015e400 msgr2=0x7f2ae00c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adeffd640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ae015e400 0x7f2ae00c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adeffd640 1 -- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ae00d1470 msgr2=0x7f2ae00c8740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adeffd640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ae00d1470 0x7f2ae00c8740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adeffd640 1 -- 172.21.15.161:0/3153468781 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ad00094d0 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adeffd640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00cf9d0 0x7f2ae00cb930 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0000c00 tx=0x7f2ad0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adcff9640 1 -- 172.21.15.161:0/3153468781 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad0039a90 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adcff9640 1 -- 172.21.15.161:0/3153468781 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ad0039bf0 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ae00c9320 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2adcff9640 1 -- 172.21.15.161:0/3153468781 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ad00433f0 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.732+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ae00c98e0 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.733+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ae00a1e00 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.733+0000 7f2adcff9640 1 -- 172.21.15.161:0/3153468781 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2ad004c3b0 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.734+0000 7f2adcff9640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2ac006b840 0x7f2ac006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:11.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.734+0000 7f2adcff9640 1 -- 172.21.15.161:0/3153468781 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f2ad00ac600 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.734+0000 7f2ad7fff640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2ac006b840 0x7f2ac006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:11.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.734+0000 7f2ad7fff640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2ac006b840 0x7f2ac006dd00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00cb370 tx=0x7f2ac8009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:11.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.738+0000 7f2adcff9640 1 -- 172.21.15.161:0/3153468781 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2ad0077160 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.908 INFO:tasks.ceph.mds.d.smithi183.stderr:2024-03-23T14:36:11.911+0000 7f579843b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T14:36:11.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.976+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f2ae004f550 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.976+0000 7f2adcff9640 1 -- 172.21.15.161:0/3153468781 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+1656 (secure 0 0 0) 0x7f2ad0034690 con 0x7f2ae00cf9d0 2024-03-23T14:36:11.972 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:11.973 INFO:teuthology.orchestra.run.smithi161.stdout:{"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":83,"monmap":{"epoch":1,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":33,"num_osds":8,"num_up_osds":8,"osd_up_since":1711204499,"num_in_osds":8,"osd_in_since":1711204491,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"unknown","count":64},{"state_name":"creating+peering","count":32},{"state_name":"active+clean","count":1}],"num_pgs":97,"num_pools":3,"num_objects":2,"data_bytes":590368,"bytes_used":225198080,"bytes_avail":772868915200,"bytes_total":773094113280,"unknown_pgs_ratio":0.65979379415512085,"inactive_pgs_ratio":0.32989689707756042},"fsmap":{"epoch":15,"id":2,"up":1,"in":1,"max":1,"by_rank":[{"filesystem_id":2,"rank":0,"name":"d","status":"up:active","gid":5235}],"up:standby":3},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-03-23T14:36:02.483292+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"c":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"d":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.978+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2ac006b840 msgr2=0x7f2ac006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.978+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2ac006b840 0x7f2ac006dd00 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2ae00cb370 tx=0x7f2ac8009190 comp rx=0 tx=0).stop 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.978+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00cf9d0 msgr2=0x7f2ae00cb930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.978+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00cf9d0 0x7f2ae00cb930 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f2ad0000c00 tx=0x7f2ad0004730 comp rx=0 tx=0).stop 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.979+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 shutdown_connections 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.979+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2ac006b840 0x7f2ac006dd00 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.979+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2ae015e400 0x7f2ae00c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.979+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2ae00d1470 0x7f2ae00c8740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.979+0000 7f2ae5a2a640 1 --2- 172.21.15.161:0/3153468781 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2ae00cf9d0 0x7f2ae00cb930 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.979+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 >> 172.21.15.161:0/3153468781 conn(0x7f2ae009d190 msgr2=0x7f2ae00b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.979+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 shutdown_connections 2024-03-23T14:36:11.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:11.979+0000 7f2ae5a2a640 1 -- 172.21.15.161:0/3153468781 wait complete. 2024-03-23T14:36:11.985 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-03-23T14:36:11.986 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:36:12.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.090+0000 7fab30bc9640 1 Processor -- start 2024-03-23T14:36:12.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.091+0000 7fab30bc9640 1 -- start start 2024-03-23T14:36:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.091+0000 7fab30bc9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fab2c145080 0x7fab2c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.091+0000 7fab30bc9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2c0cf4f0 con 0x7fab2c1601f0 2024-03-23T14:36:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.091+0000 7fab30bc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fab2c0cef30 con 0x7fab2c1607d0 2024-03-23T14:36:12.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.091+0000 7fab30bc9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fab2c071010 con 0x7fab2c145080 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.092+0000 7fab2a59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fab2c145080 0x7fab2c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.092+0000 7fab2ad9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fab2c1601f0 0x7fab2c15c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45466/0 (socket says 172.21.15.161:45466) 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.092+0000 7fab2ad9d640 1 -- 172.21.15.161:0/37044623 learned_addr learned my addr 172.21.15.161:0/37044623 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.092+0000 7fab2a59c640 1 -- 172.21.15.161:0/37044623 >> v1:172.21.15.183:6789/0 conn(0x7fab2c1607d0 legacy=0x7fab2c15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.092+0000 7fab2a59c640 1 -- 172.21.15.161:0/37044623 >> v1:172.21.15.161:6789/0 conn(0x7fab2c1601f0 legacy=0x7fab2c15c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.092+0000 7fab2a59c640 1 -- 172.21.15.161:0/37044623 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab2c0cf210 con 0x7fab2c145080 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.092+0000 7fab29d9b640 1 -- 172.21.15.161:0/37044623 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4259951593 0 0) 0x7fab2c0cf4f0 con 0x7fab2c1601f0 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab2a59c640 1 --2- 172.21.15.161:0/37044623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c145080 0x7fab2c072a00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0097a0 tx=0x7fab1c02b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48c48a971bbca5e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab29d9b640 1 -- 172.21.15.161:0/37044623 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c031830 con 0x7fab2c145080 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab29d9b640 1 -- 172.21.15.161:0/37044623 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab1c031990 con 0x7fab2c145080 2024-03-23T14:36:12.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab29d9b640 1 -- 172.21.15.161:0/37044623 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab1c031d20 con 0x7fab2c145080 2024-03-23T14:36:12.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab30bc9640 1 -- 172.21.15.161:0/37044623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c145080 msgr2=0x7fab2c072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:12.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab30bc9640 1 --2- 172.21.15.161:0/37044623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c145080 0x7fab2c072a00 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fab1c0097a0 tx=0x7fab1c02b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:12.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab30bc9640 1 -- 172.21.15.161:0/37044623 shutdown_connections 2024-03-23T14:36:12.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab30bc9640 1 --2- 172.21.15.161:0/37044623 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c145080 0x7fab2c072a00 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:12.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab30bc9640 1 -- 172.21.15.161:0/37044623 >> 172.21.15.161:0/37044623 conn(0x7fab2c0b17f0 msgr2=0x7fab2c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:12.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab30bc9640 1 -- 172.21.15.161:0/37044623 shutdown_connections 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.093+0000 7fab30bc9640 1 -- 172.21.15.161:0/37044623 wait complete. 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab30bc9640 1 Processor -- start 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab30bc9640 1 -- start start 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab30bc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab2c145080 0x7fab2c159670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab30bc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c1601f0 0x7fab2c139890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab30bc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab2c1607d0 0x7fab2c15e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab30bc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c13cb40 con 0x7fab2c1601f0 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab30bc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fab2c13bf30 con 0x7fab2c145080 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab30bc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fab2c13c210 con 0x7fab2c1607d0 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab23fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c1601f0 0x7fab2c139890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab23fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c1601f0 0x7fab2c139890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36506/0 (socket says 172.21.15.161:36506) 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab23fff640 1 -- 172.21.15.161:0/2987473665 learned_addr learned my addr 172.21.15.161:0/2987473665 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.095+0000 7fab2a59c640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab2c145080 0x7fab2c159670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:12.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab2ad9d640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab2c1607d0 0x7fab2c15e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab23fff640 1 -- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab2c1607d0 msgr2=0x7fab2c15e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab23fff640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab2c1607d0 0x7fab2c15e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab23fff640 1 -- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab2c145080 msgr2=0x7fab2c159670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab23fff640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab2c145080 0x7fab2c159670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab23fff640 1 -- 172.21.15.161:0/2987473665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab1c009400 con 0x7fab2c1601f0 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab23fff640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c1601f0 0x7fab2c139890 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c3c0 tx=0x7fab1400c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab22ffd640 1 -- 172.21.15.161:0/2987473665 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab140052d0 con 0x7fab2c1601f0 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab22ffd640 1 -- 172.21.15.161:0/2987473665 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fab14005450 con 0x7fab2c1601f0 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab22ffd640 1 -- 172.21.15.161:0/2987473665 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab14003830 con 0x7fab2c1601f0 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab2c13c4f0 con 0x7fab2c1601f0 2024-03-23T14:36:12.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.096+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab2c159bb0 con 0x7fab2c1601f0 2024-03-23T14:36:12.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.097+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab2c0b2db0 con 0x7fab2c1601f0 2024-03-23T14:36:12.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.097+0000 7fab22ffd640 1 -- 172.21.15.161:0/2987473665 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fab14003990 con 0x7fab2c1601f0 2024-03-23T14:36:12.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.098+0000 7fab22ffd640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fab0c06b810 0x7fab0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:12.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.098+0000 7fab22ffd640 1 -- 172.21.15.161:0/2987473665 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fab14089670 con 0x7fab2c1601f0 2024-03-23T14:36:12.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.098+0000 7fab2a59c640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fab0c06b810 0x7fab0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:12.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.098+0000 7fab2a59c640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fab0c06b810 0x7fab0c06dcd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab1c036000 tx=0x7fab1c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:12.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.102+0000 7fab22ffd640 1 -- 172.21.15.161:0/2987473665 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fab14054140 con 0x7fab2c1601f0 2024-03-23T14:36:12.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.269+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fab2c13cc80 con 0x7fab0c06b810 2024-03-23T14:36:12.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.274+0000 7fab22ffd640 1 -- 172.21.15.161:0/2987473665 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+239826 (secure 0 0 0) 0x7fab2c13cc80 con 0x7fab0c06b810 2024-03-23T14:36:12.270 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:12.274 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:36:12.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fab0c06b810 msgr2=0x7fab0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fab0c06b810 0x7fab0c06dcd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab1c036000 tx=0x7fab1c03c040 comp rx=0 tx=0).stop 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c1601f0 msgr2=0x7fab2c139890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c1601f0 0x7fab2c139890 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fab1400c3c0 tx=0x7fab1400c880 comp rx=0 tx=0).stop 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 shutdown_connections 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fab0c06b810 0x7fab0c06dcd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fab2c1607d0 0x7fab2c15e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fab2c1601f0 0x7fab2c139890 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 --2- 172.21.15.161:0/2987473665 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fab2c145080 0x7fab2c159670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 >> 172.21.15.161:0/2987473665 conn(0x7fab2c0b17f0 msgr2=0x7fab2c06d850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 shutdown_connections 2024-03-23T14:36:12.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:12.277+0000 7fab30bc9640 1 -- 172.21.15.161:0/2987473665 wait complete. 2024-03-23T14:36:12.284 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":61,"stamp":"2024-03-23T14:36:10.484585+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":66,"acting":66,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":4,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":219920,"kb_used_data":3748,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754800,"statfs":{"total":773094113280,"available":772868915200,"internally_reserved":0,"allocated":3837952,"data_stored":2181477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.568230"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921146+0000","last_change":"2024-03-23T14:36:04.921146+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921146+0000","last_undegraded":"2024-03-23T14:36:04.921146+0000","last_fullsized":"2024-03-23T14:36:04.921146+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919587+0000","last_change":"2024-03-23T14:36:04.919587+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919587+0000","last_undegraded":"2024-03-23T14:36:04.919587+0000","last_fullsized":"2024-03-23T14:36:04.919587+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922002+0000","last_change":"2024-03-23T14:36:04.922002+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922002+0000","last_undegraded":"2024-03-23T14:36:04.922002+0000","last_fullsized":"2024-03-23T14:36:04.922002+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920050+0000","last_change":"2024-03-23T14:36:04.920050+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920050+0000","last_undegraded":"2024-03-23T14:36:04.920050+0000","last_fullsized":"2024-03-23T14:36:04.920050+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919319+0000","last_change":"2024-03-23T14:36:04.919319+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919319+0000","last_undegraded":"2024-03-23T14:36:04.919319+0000","last_fullsized":"2024-03-23T14:36:04.919319+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918554+0000","last_change":"2024-03-23T14:36:04.918554+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918554+0000","last_undegraded":"2024-03-23T14:36:04.918554+0000","last_fullsized":"2024-03-23T14:36:04.918554+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921218+0000","last_change":"2024-03-23T14:36:04.921218+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921218+0000","last_undegraded":"2024-03-23T14:36:04.921218+0000","last_fullsized":"2024-03-23T14:36:04.921218+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919010+0000","last_change":"2024-03-23T14:36:04.919010+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919010+0000","last_undegraded":"2024-03-23T14:36:04.919010+0000","last_fullsized":"2024-03-23T14:36:04.919010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920263+0000","last_change":"2024-03-23T14:36:04.920263+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920263+0000","last_undegraded":"2024-03-23T14:36:04.920263+0000","last_fullsized":"2024-03-23T14:36:04.920263+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919149+0000","last_change":"2024-03-23T14:36:04.919149+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919149+0000","last_undegraded":"2024-03-23T14:36:04.919149+0000","last_fullsized":"2024-03-23T14:36:04.919149+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921107+0000","last_change":"2024-03-23T14:36:04.921107+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921107+0000","last_undegraded":"2024-03-23T14:36:04.921107+0000","last_fullsized":"2024-03-23T14:36:04.921107+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920278+0000","last_change":"2024-03-23T14:36:04.920278+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920278+0000","last_undegraded":"2024-03-23T14:36:04.920278+0000","last_fullsized":"2024-03-23T14:36:04.920278+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918667+0000","last_change":"2024-03-23T14:36:04.918667+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918667+0000","last_undegraded":"2024-03-23T14:36:04.918667+0000","last_fullsized":"2024-03-23T14:36:04.918667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918094+0000","last_change":"2024-03-23T14:36:04.918094+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918094+0000","last_undegraded":"2024-03-23T14:36:04.918094+0000","last_fullsized":"2024-03-23T14:36:04.918094+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921589+0000","last_change":"2024-03-23T14:36:04.921589+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921589+0000","last_undegraded":"2024-03-23T14:36:04.921589+0000","last_fullsized":"2024-03-23T14:36:04.921589+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920831+0000","last_change":"2024-03-23T14:36:04.920831+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920831+0000","last_undegraded":"2024-03-23T14:36:04.920831+0000","last_fullsized":"2024-03-23T14:36:04.920831+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918060+0000","last_change":"2024-03-23T14:36:04.918060+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918060+0000","last_undegraded":"2024-03-23T14:36:04.918060+0000","last_fullsized":"2024-03-23T14:36:04.918060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919674+0000","last_change":"2024-03-23T14:36:04.919674+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919674+0000","last_undegraded":"2024-03-23T14:36:04.919674+0000","last_fullsized":"2024-03-23T14:36:04.919674+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.919612+0000","last_change":"2024-03-23T14:36:04.919612+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.919612+0000","last_undegraded":"2024-03-23T14:36:04.919612+0000","last_fullsized":"2024-03-23T14:36:04.919612+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918594+0000","last_change":"2024-03-23T14:36:04.918594+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918594+0000","last_undegraded":"2024-03-23T14:36:04.918594+0000","last_fullsized":"2024-03-23T14:36:04.918594+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918956+0000","last_change":"2024-03-23T14:36:04.918956+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918956+0000","last_undegraded":"2024-03-23T14:36:04.918956+0000","last_fullsized":"2024-03-23T14:36:04.918956+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921160+0000","last_change":"2024-03-23T14:36:04.921160+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921160+0000","last_undegraded":"2024-03-23T14:36:04.921160+0000","last_fullsized":"2024-03-23T14:36:04.921160+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.920128+0000","last_change":"2024-03-23T14:36:04.920128+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.920128+0000","last_undegraded":"2024-03-23T14:36:04.920128+0000","last_fullsized":"2024-03-23T14:36:04.920128+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922010+0000","last_change":"2024-03-23T14:36:04.922010+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922010+0000","last_undegraded":"2024-03-23T14:36:04.922010+0000","last_fullsized":"2024-03-23T14:36:04.922010+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917607+0000","last_change":"2024-03-23T14:36:04.917607+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917607+0000","last_undegraded":"2024-03-23T14:36:04.917607+0000","last_fullsized":"2024-03-23T14:36:04.917607+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918696+0000","last_change":"2024-03-23T14:36:04.918696+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918696+0000","last_undegraded":"2024-03-23T14:36:04.918696+0000","last_fullsized":"2024-03-23T14:36:04.918696+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917667+0000","last_change":"2024-03-23T14:36:04.917667+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917667+0000","last_undegraded":"2024-03-23T14:36:04.917667+0000","last_fullsized":"2024-03-23T14:36:04.917667+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918505+0000","last_change":"2024-03-23T14:36:04.918505+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918505+0000","last_undegraded":"2024-03-23T14:36:04.918505+0000","last_fullsized":"2024-03-23T14:36:04.918505+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.917659+0000","last_change":"2024-03-23T14:36:04.917659+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.917659+0000","last_undegraded":"2024-03-23T14:36:04.917659+0000","last_fullsized":"2024-03-23T14:36:04.917659+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.918561+0000","last_change":"2024-03-23T14:36:04.918561+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.918561+0000","last_undegraded":"2024-03-23T14:36:04.918561+0000","last_fullsized":"2024-03-23T14:36:04.918561+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.922105+0000","last_change":"2024-03-23T14:36:04.922105+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.922105+0000","last_undegraded":"2024-03-23T14:36:04.922105+0000","last_fullsized":"2024-03-23T14:36:04.922105+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:05.914757+0000","last_change":"2024-03-23T14:36:05.914757+0000","last_active":"2024-03-23T14:36:05.914757+0000","last_peered":"2024-03-23T14:36:05.914757+0000","last_clean":"2024-03-23T14:36:05.914757+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:05.914757+0000","last_undegraded":"2024-03-23T14:36:05.914757+0000","last_fullsized":"2024-03-23T14:36:05.914757+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":1,"reported_epoch":30,"state":"creating+peering","last_fresh":"2024-03-23T14:36:04.921452+0000","last_change":"2024-03-23T14:36:04.921452+0000","last_active":"2024-03-23T14:36:04.910778+0000","last_peered":"2024-03-23T14:36:04.910778+0000","last_clean":"2024-03-23T14:36:04.910778+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:04.921452+0000","last_undegraded":"2024-03-23T14:36:04.921452+0000","last_fullsized":"2024-03-23T14:36:04.921452+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":186,"reported_epoch":30,"state":"active+clean","last_fresh":"2024-03-23T14:36:04.922822+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:36:04.922822+0000","last_peered":"2024-03-23T14:36:04.922822+0000","last_clean":"2024-03-23T14:36:04.922822+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:36:04.922822+0000","last_undegraded":"2024-03-23T14:36:04.922822+0000","last_fullsized":"2024-03-23T14:36:04.922822+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27916,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343924,"statfs":{"total":96636764160,"available":96608178176,"internally_reserved":0,"allocated":917504,"data_stored":711825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574864,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27336,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344504,"statfs":{"total":96636764160,"available":96608772096,"internally_reserved":0,"allocated":323584,"data_stored":121457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574865,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":328,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":335872,"data_stored":127274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574866,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27348,"kb_used_data":316,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344492,"statfs":{"total":96636764160,"available":96608759808,"internally_reserved":0,"allocated":323584,"data_stored":121457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574865,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27928,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343912,"statfs":{"total":96636764160,"available":96608165888,"internally_reserved":0,"allocated":929792,"data_stored":717642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:36:15.288 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:36:15.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.404+0000 7f7cdfb3e640 1 Processor -- start 2024-03-23T14:36:15.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.405+0000 7f7cdfb3e640 1 -- start start 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.405+0000 7f7cdfb3e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7cd815e400 0x7f7cd80cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.405+0000 7f7cdfb3e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7cd80cf9d0 0x7f7cd80d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdfb3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cd80d0460 con 0x7f7cd80d1470 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdfb3e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7cd80d1050 con 0x7f7cd815e400 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdfb3e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f7cd80d06b0 con 0x7f7cd80cf9d0 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdd8da640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7cd815e400 0x7f7cd80cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdd8da640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7cd815e400 0x7f7cd80cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36518/0 (socket says 172.21.15.161:36518) 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdd8da640 1 -- 172.21.15.161:0/1961290679 learned_addr learned my addr 172.21.15.161:0/1961290679 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:15.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdd0d9640 1 --2- 172.21.15.161:0/1961290679 >> v2:172.21.15.183:3300/0 conn(0x7f7cd80cf9d0 0x7f7cd80d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdd8da640 1 -- 172.21.15.161:0/1961290679 >> v2:172.21.15.183:3300/0 conn(0x7f7cd80cf9d0 msgr2=0x7f7cd80d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:15.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdd8da640 1 --2- 172.21.15.161:0/1961290679 >> v2:172.21.15.183:3300/0 conn(0x7f7cd80cf9d0 0x7f7cd80d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdd8da640 1 -- 172.21.15.161:0/1961290679 >> v1:172.21.15.183:6789/0 conn(0x7f7cd80d1470 legacy=0x7f7cd80b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.406+0000 7f7cdd8da640 1 -- 172.21.15.161:0/1961290679 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd80d0990 con 0x7f7cd815e400 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.407+0000 7f7cdd8da640 1 --2- 172.21.15.161:0/1961290679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd815e400 0x7f7cd80cf490 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80b6a20 tx=0x7f7cd402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3a17a950e6a7384 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.407+0000 7f7cdc8d8640 1 -- 172.21.15.161:0/1961290679 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4030860 con 0x7f7cd815e400 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.407+0000 7f7cdc8d8640 1 -- 172.21.15.161:0/1961290679 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd40309c0 con 0x7f7cd815e400 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.407+0000 7f7cdc8d8640 1 -- 172.21.15.161:0/1961290679 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4030d50 con 0x7f7cd815e400 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.407+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/1961290679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd815e400 msgr2=0x7f7cd80cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.407+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/1961290679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd815e400 0x7f7cd80cf490 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80b6a20 tx=0x7f7cd402b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.408+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/1961290679 shutdown_connections 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.408+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/1961290679 >> v2:172.21.15.183:3300/0 conn(0x7f7cd80cf9d0 0x7f7cd80d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.408+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/1961290679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd815e400 0x7f7cd80cf490 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.408+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/1961290679 >> 172.21.15.161:0/1961290679 conn(0x7f7cd809d190 msgr2=0x7f7cd80d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:15.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.408+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/1961290679 shutdown_connections 2024-03-23T14:36:15.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.408+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/1961290679 wait complete. 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.409+0000 7f7cdfb3e640 1 Processor -- start 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.409+0000 7f7cdfb3e640 1 -- start start 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdfb3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7cd80cf9d0 0x7f7cd80cb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdfb3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd80d1470 0x7f7cd80c8740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdfb3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd815e400 0x7f7cd80c9e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdfb3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd80ca340 con 0x7f7cd80d1470 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdfb3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd80cbff0 con 0x7f7cd815e400 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdfb3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd80c9040 con 0x7f7cd80cf9d0 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd80d1470 0x7f7cd80c8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd80d1470 0x7f7cd80c8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36530/0 (socket says 172.21.15.161:36530) 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 -- 172.21.15.161:0/241059224 learned_addr learned my addr 172.21.15.161:0/241059224 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:15.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd8da640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7cd80cf9d0 0x7f7cd80cb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cde0db640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd815e400 0x7f7cd80c9e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 -- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7cd80cf9d0 msgr2=0x7f7cd80cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7cd80cf9d0 0x7f7cd80cb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 -- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd815e400 msgr2=0x7f7cd80c9e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd815e400 0x7f7cd80c9e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 -- 172.21.15.161:0/241059224 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd40094d0 con 0x7f7cd80d1470 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdd0d9640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd80d1470 0x7f7cd80c8740 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800e9b0 tx=0x7f7cc800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cce7fc640 1 -- 172.21.15.161:0/241059224 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8009670 con 0x7f7cd80d1470 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cce7fc640 1 -- 172.21.15.161:0/241059224 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc800c9a0 con 0x7f7cd80d1470 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd80c9320 con 0x7f7cd80d1470 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cce7fc640 1 -- 172.21.15.161:0/241059224 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc800cd50 con 0x7f7cd80d1470 2024-03-23T14:36:15.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.410+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd80c98e0 con 0x7f7cd80d1470 2024-03-23T14:36:15.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.411+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd80c9aa0 con 0x7f7cd80d1470 2024-03-23T14:36:15.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.412+0000 7f7cce7fc640 1 -- 172.21.15.161:0/241059224 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7cc8013470 con 0x7f7cd80d1470 2024-03-23T14:36:15.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.412+0000 7f7cce7fc640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7cbc06b840 0x7f7cbc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.412+0000 7f7cce7fc640 1 -- 172.21.15.161:0/241059224 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f7cc808a2a0 con 0x7f7cd80d1470 2024-03-23T14:36:15.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.412+0000 7f7cdd8da640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7cbc06b840 0x7f7cbc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.413+0000 7f7cdd8da640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7cbc06b840 0x7f7cbc06dd00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4036000 tx=0x7f7cd4008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:15.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.417+0000 7f7cce7fc640 1 -- 172.21.15.161:0/241059224 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7cc8054e00 con 0x7f7cd80d1470 2024-03-23T14:36:15.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.586+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7cd80c91b0 con 0x7f7cbc06b840 2024-03-23T14:36:15.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.592+0000 7f7cce7fc640 1 -- 172.21.15.161:0/241059224 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251951 (secure 0 0 0) 0x7f7cd80c91b0 con 0x7f7cbc06b840 2024-03-23T14:36:15.591 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:15.592 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:36:15.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7cbc06b840 msgr2=0x7f7cbc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7cbc06b840 0x7f7cbc06dd00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4036000 tx=0x7f7cd4008030 comp rx=0 tx=0).stop 2024-03-23T14:36:15.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd80d1470 msgr2=0x7f7cd80c8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd80d1470 0x7f7cd80c8740 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f7cc800e9b0 tx=0x7f7cc800ee70 comp rx=0 tx=0).stop 2024-03-23T14:36:15.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 shutdown_connections 2024-03-23T14:36:15.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7cbc06b840 0x7f7cbc06dd00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7cd815e400 0x7f7cd80c9e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7cd80d1470 0x7f7cd80c8740 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 --2- 172.21.15.161:0/241059224 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7cd80cf9d0 0x7f7cd80cb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 >> 172.21.15.161:0/241059224 conn(0x7f7cd809d190 msgr2=0x7f7cd80b0a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:15.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 shutdown_connections 2024-03-23T14:36:15.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.595+0000 7f7cdfb3e640 1 -- 172.21.15.161:0/241059224 wait complete. 2024-03-23T14:36:15.602 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":63,"stamp":"2024-03-23T14:36:14.485292+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":176,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":220328,"kb_used_data":4136,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754754392,"statfs":{"total":773094113280,"available":772868497408,"internally_reserved":0,"allocated":4235264,"data_stored":2358484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":37,"apply_latency_ms":37,"commit_latency_ns":37000000,"apply_latency_ns":37000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001356"},"pg_stats":[{"pgid":"5.3f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302200+0000","last_change":"2024-03-23T14:36:06.955690+0000","last_active":"2024-03-23T14:36:08.302200+0000","last_peered":"2024-03-23T14:36:08.302200+0000","last_clean":"2024-03-23T14:36:08.302200+0000","last_became_active":"2024-03-23T14:36:06.955183+0000","last_became_peered":"2024-03-23T14:36:06.955183+0000","last_unstale":"2024-03-23T14:36:08.302200+0000","last_undegraded":"2024-03-23T14:36:08.302200+0000","last_fullsized":"2024-03-23T14:36:08.302200+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:14:30.764439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.3e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.303056+0000","last_change":"2024-03-23T14:36:06.940597+0000","last_active":"2024-03-23T14:36:08.303056+0000","last_peered":"2024-03-23T14:36:08.303056+0000","last_clean":"2024-03-23T14:36:08.303056+0000","last_became_active":"2024-03-23T14:36:06.940279+0000","last_became_peered":"2024-03-23T14:36:06.940279+0000","last_unstale":"2024-03-23T14:36:08.303056+0000","last_undegraded":"2024-03-23T14:36:08.303056+0000","last_fullsized":"2024-03-23T14:36:08.303056+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:54:12.738117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.3d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302030+0000","last_change":"2024-03-23T14:36:06.955744+0000","last_active":"2024-03-23T14:36:08.302030+0000","last_peered":"2024-03-23T14:36:08.302030+0000","last_clean":"2024-03-23T14:36:08.302030+0000","last_became_active":"2024-03-23T14:36:06.955462+0000","last_became_peered":"2024-03-23T14:36:06.955462+0000","last_unstale":"2024-03-23T14:36:08.302030+0000","last_undegraded":"2024-03-23T14:36:08.302030+0000","last_fullsized":"2024-03-23T14:36:08.302030+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:17:22.437212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929746+0000","last_change":"2024-03-23T14:36:06.951068+0000","last_active":"2024-03-23T14:36:07.929746+0000","last_peered":"2024-03-23T14:36:07.929746+0000","last_clean":"2024-03-23T14:36:07.929746+0000","last_became_active":"2024-03-23T14:36:06.950803+0000","last_became_peered":"2024-03-23T14:36:06.950803+0000","last_unstale":"2024-03-23T14:36:07.929746+0000","last_undegraded":"2024-03-23T14:36:07.929746+0000","last_fullsized":"2024-03-23T14:36:07.929746+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:35.559868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302251+0000","last_change":"2024-03-23T14:36:06.938937+0000","last_active":"2024-03-23T14:36:08.302251+0000","last_peered":"2024-03-23T14:36:08.302251+0000","last_clean":"2024-03-23T14:36:08.302251+0000","last_became_active":"2024-03-23T14:36:06.938686+0000","last_became_peered":"2024-03-23T14:36:06.938686+0000","last_unstale":"2024-03-23T14:36:08.302251+0000","last_undegraded":"2024-03-23T14:36:08.302251+0000","last_fullsized":"2024-03-23T14:36:08.302251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:55.220848+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.3a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302714+0000","last_change":"2024-03-23T14:36:06.942551+0000","last_active":"2024-03-23T14:36:08.302714+0000","last_peered":"2024-03-23T14:36:08.302714+0000","last_clean":"2024-03-23T14:36:08.302714+0000","last_became_active":"2024-03-23T14:36:06.942282+0000","last_became_peered":"2024-03-23T14:36:06.942282+0000","last_unstale":"2024-03-23T14:36:08.302714+0000","last_undegraded":"2024-03-23T14:36:08.302714+0000","last_fullsized":"2024-03-23T14:36:08.302714+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:17:22.437212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.39","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195259+0000","last_change":"2024-03-23T14:36:06.946713+0000","last_active":"2024-03-23T14:36:08.195259+0000","last_peered":"2024-03-23T14:36:08.195259+0000","last_clean":"2024-03-23T14:36:08.195259+0000","last_became_active":"2024-03-23T14:36:06.946445+0000","last_became_peered":"2024-03-23T14:36:06.946445+0000","last_unstale":"2024-03-23T14:36:08.195259+0000","last_undegraded":"2024-03-23T14:36:08.195259+0000","last_fullsized":"2024-03-23T14:36:08.195259+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:29:06.240340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.38","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195197+0000","last_change":"2024-03-23T14:36:06.945478+0000","last_active":"2024-03-23T14:36:08.195197+0000","last_peered":"2024-03-23T14:36:08.195197+0000","last_clean":"2024-03-23T14:36:08.195197+0000","last_became_active":"2024-03-23T14:36:06.945184+0000","last_became_peered":"2024-03-23T14:36:06.945184+0000","last_unstale":"2024-03-23T14:36:08.195197+0000","last_undegraded":"2024-03-23T14:36:08.195197+0000","last_fullsized":"2024-03-23T14:36:08.195197+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:35.559868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.37","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930754+0000","last_change":"2024-03-23T14:36:06.950137+0000","last_active":"2024-03-23T14:36:07.930754+0000","last_peered":"2024-03-23T14:36:07.930754+0000","last_clean":"2024-03-23T14:36:07.930754+0000","last_became_active":"2024-03-23T14:36:06.949814+0000","last_became_peered":"2024-03-23T14:36:06.949814+0000","last_unstale":"2024-03-23T14:36:07.930754+0000","last_undegraded":"2024-03-23T14:36:07.930754+0000","last_fullsized":"2024-03-23T14:36:07.930754+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:29:06.240340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.36","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929808+0000","last_change":"2024-03-23T14:36:06.943836+0000","last_active":"2024-03-23T14:36:07.929808+0000","last_peered":"2024-03-23T14:36:07.929808+0000","last_clean":"2024-03-23T14:36:07.929808+0000","last_became_active":"2024-03-23T14:36:06.943540+0000","last_became_peered":"2024-03-23T14:36:06.943540+0000","last_unstale":"2024-03-23T14:36:07.929808+0000","last_undegraded":"2024-03-23T14:36:07.929808+0000","last_fullsized":"2024-03-23T14:36:07.929808+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:29:06.240340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.35","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195391+0000","last_change":"2024-03-23T14:36:06.952563+0000","last_active":"2024-03-23T14:36:08.195391+0000","last_peered":"2024-03-23T14:36:08.195391+0000","last_clean":"2024-03-23T14:36:08.195391+0000","last_became_active":"2024-03-23T14:36:06.949983+0000","last_became_peered":"2024-03-23T14:36:06.949983+0000","last_unstale":"2024-03-23T14:36:08.195391+0000","last_undegraded":"2024-03-23T14:36:08.195391+0000","last_fullsized":"2024-03-23T14:36:08.195391+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:17:22.437212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.34","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302149+0000","last_change":"2024-03-23T14:36:06.945429+0000","last_active":"2024-03-23T14:36:08.302149+0000","last_peered":"2024-03-23T14:36:08.302149+0000","last_clean":"2024-03-23T14:36:08.302149+0000","last_became_active":"2024-03-23T14:36:06.944311+0000","last_became_peered":"2024-03-23T14:36:06.944311+0000","last_unstale":"2024-03-23T14:36:08.302149+0000","last_undegraded":"2024-03-23T14:36:08.302149+0000","last_fullsized":"2024-03-23T14:36:08.302149+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:14:50.221562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.33","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302374+0000","last_change":"2024-03-23T14:36:06.938296+0000","last_active":"2024-03-23T14:36:08.302374+0000","last_peered":"2024-03-23T14:36:08.302374+0000","last_clean":"2024-03-23T14:36:08.302374+0000","last_became_active":"2024-03-23T14:36:06.938006+0000","last_became_peered":"2024-03-23T14:36:06.938006+0000","last_unstale":"2024-03-23T14:36:08.302374+0000","last_undegraded":"2024-03-23T14:36:08.302374+0000","last_fullsized":"2024-03-23T14:36:08.302374+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:08:48.906370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.32","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302768+0000","last_change":"2024-03-23T14:36:06.935335+0000","last_active":"2024-03-23T14:36:08.302768+0000","last_peered":"2024-03-23T14:36:08.302768+0000","last_clean":"2024-03-23T14:36:08.302768+0000","last_became_active":"2024-03-23T14:36:06.935033+0000","last_became_peered":"2024-03-23T14:36:06.935033+0000","last_unstale":"2024-03-23T14:36:08.302768+0000","last_undegraded":"2024-03-23T14:36:08.302768+0000","last_fullsized":"2024-03-23T14:36:08.302768+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:14:50.221562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.31","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301831+0000","last_change":"2024-03-23T14:36:06.947568+0000","last_active":"2024-03-23T14:36:08.301831+0000","last_peered":"2024-03-23T14:36:08.301831+0000","last_clean":"2024-03-23T14:36:08.301831+0000","last_became_active":"2024-03-23T14:36:06.947297+0000","last_became_peered":"2024-03-23T14:36:06.947297+0000","last_unstale":"2024-03-23T14:36:08.301831+0000","last_undegraded":"2024-03-23T14:36:08.301831+0000","last_fullsized":"2024-03-23T14:36:08.301831+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:29:22.028913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.30","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195577+0000","last_change":"2024-03-23T14:36:06.946305+0000","last_active":"2024-03-23T14:36:08.195577+0000","last_peered":"2024-03-23T14:36:08.195577+0000","last_clean":"2024-03-23T14:36:08.195577+0000","last_became_active":"2024-03-23T14:36:06.946047+0000","last_became_peered":"2024-03-23T14:36:06.946047+0000","last_unstale":"2024-03-23T14:36:08.195577+0000","last_undegraded":"2024-03-23T14:36:08.195577+0000","last_fullsized":"2024-03-23T14:36:08.195577+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:21.302001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195555+0000","last_change":"2024-03-23T14:36:06.948921+0000","last_active":"2024-03-23T14:36:08.195555+0000","last_peered":"2024-03-23T14:36:08.195555+0000","last_clean":"2024-03-23T14:36:08.195555+0000","last_became_active":"2024-03-23T14:36:06.948654+0000","last_became_peered":"2024-03-23T14:36:06.948654+0000","last_unstale":"2024-03-23T14:36:08.195555+0000","last_undegraded":"2024-03-23T14:36:08.195555+0000","last_fullsized":"2024-03-23T14:36:08.195555+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:43:54.788598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.2e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.303121+0000","last_change":"2024-03-23T14:36:06.946815+0000","last_active":"2024-03-23T14:36:08.303121+0000","last_peered":"2024-03-23T14:36:08.303121+0000","last_clean":"2024-03-23T14:36:08.303121+0000","last_became_active":"2024-03-23T14:36:06.946377+0000","last_became_peered":"2024-03-23T14:36:06.946377+0000","last_unstale":"2024-03-23T14:36:08.303121+0000","last_undegraded":"2024-03-23T14:36:08.303121+0000","last_fullsized":"2024-03-23T14:36:08.303121+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:35:50.726321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302952+0000","last_change":"2024-03-23T14:36:06.946061+0000","last_active":"2024-03-23T14:36:08.302952+0000","last_peered":"2024-03-23T14:36:08.302952+0000","last_clean":"2024-03-23T14:36:08.302952+0000","last_became_active":"2024-03-23T14:36:06.945809+0000","last_became_peered":"2024-03-23T14:36:06.945809+0000","last_unstale":"2024-03-23T14:36:08.302952+0000","last_undegraded":"2024-03-23T14:36:08.302952+0000","last_fullsized":"2024-03-23T14:36:08.302952+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:21.302001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.2c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302571+0000","last_change":"2024-03-23T14:36:06.942995+0000","last_active":"2024-03-23T14:36:08.302571+0000","last_peered":"2024-03-23T14:36:08.302571+0000","last_clean":"2024-03-23T14:36:08.302571+0000","last_became_active":"2024-03-23T14:36:06.942721+0000","last_became_peered":"2024-03-23T14:36:06.942721+0000","last_unstale":"2024-03-23T14:36:08.302571+0000","last_undegraded":"2024-03-23T14:36:08.302571+0000","last_fullsized":"2024-03-23T14:36:08.302571+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:24.406986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930350+0000","last_change":"2024-03-23T14:36:06.947891+0000","last_active":"2024-03-23T14:36:07.930350+0000","last_peered":"2024-03-23T14:36:07.930350+0000","last_clean":"2024-03-23T14:36:07.930350+0000","last_became_active":"2024-03-23T14:36:06.947534+0000","last_became_peered":"2024-03-23T14:36:06.947534+0000","last_unstale":"2024-03-23T14:36:07.930350+0000","last_undegraded":"2024-03-23T14:36:07.930350+0000","last_fullsized":"2024-03-23T14:36:07.930350+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:09:27.643193+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.2a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930287+0000","last_change":"2024-03-23T14:36:06.935323+0000","last_active":"2024-03-23T14:36:07.930287+0000","last_peered":"2024-03-23T14:36:07.930287+0000","last_clean":"2024-03-23T14:36:07.930287+0000","last_became_active":"2024-03-23T14:36:06.934872+0000","last_became_peered":"2024-03-23T14:36:06.934872+0000","last_unstale":"2024-03-23T14:36:07.930287+0000","last_undegraded":"2024-03-23T14:36:07.930287+0000","last_fullsized":"2024-03-23T14:36:07.930287+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:37.293562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.29","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.928715+0000","last_change":"2024-03-23T14:36:06.941340+0000","last_active":"2024-03-23T14:36:07.928715+0000","last_peered":"2024-03-23T14:36:07.928715+0000","last_clean":"2024-03-23T14:36:07.928715+0000","last_became_active":"2024-03-23T14:36:06.941065+0000","last_became_peered":"2024-03-23T14:36:06.941065+0000","last_unstale":"2024-03-23T14:36:07.928715+0000","last_undegraded":"2024-03-23T14:36:07.928715+0000","last_fullsized":"2024-03-23T14:36:07.928715+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:23:52.993846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.28","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301757+0000","last_change":"2024-03-23T14:36:06.941229+0000","last_active":"2024-03-23T14:36:08.301757+0000","last_peered":"2024-03-23T14:36:08.301757+0000","last_clean":"2024-03-23T14:36:08.301757+0000","last_became_active":"2024-03-23T14:36:06.940948+0000","last_became_peered":"2024-03-23T14:36:06.940948+0000","last_unstale":"2024-03-23T14:36:08.301757+0000","last_undegraded":"2024-03-23T14:36:08.301757+0000","last_fullsized":"2024-03-23T14:36:08.301757+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:29:06.240340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.27","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929892+0000","last_change":"2024-03-23T14:36:06.942256+0000","last_active":"2024-03-23T14:36:07.929892+0000","last_peered":"2024-03-23T14:36:07.929892+0000","last_clean":"2024-03-23T14:36:07.929892+0000","last_became_active":"2024-03-23T14:36:06.941911+0000","last_became_peered":"2024-03-23T14:36:06.941911+0000","last_unstale":"2024-03-23T14:36:07.929892+0000","last_undegraded":"2024-03-23T14:36:07.929892+0000","last_fullsized":"2024-03-23T14:36:07.929892+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:29:22.028913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.26","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.303135+0000","last_change":"2024-03-23T14:36:06.943592+0000","last_active":"2024-03-23T14:36:08.303135+0000","last_peered":"2024-03-23T14:36:08.303135+0000","last_clean":"2024-03-23T14:36:08.303135+0000","last_became_active":"2024-03-23T14:36:06.943338+0000","last_became_peered":"2024-03-23T14:36:06.943338+0000","last_unstale":"2024-03-23T14:36:08.303135+0000","last_undegraded":"2024-03-23T14:36:08.303135+0000","last_fullsized":"2024-03-23T14:36:08.303135+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:01:56.763915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.25","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930076+0000","last_change":"2024-03-23T14:36:06.950458+0000","last_active":"2024-03-23T14:36:07.930076+0000","last_peered":"2024-03-23T14:36:07.930076+0000","last_clean":"2024-03-23T14:36:07.930076+0000","last_became_active":"2024-03-23T14:36:06.950187+0000","last_became_peered":"2024-03-23T14:36:06.950187+0000","last_unstale":"2024-03-23T14:36:07.930076+0000","last_undegraded":"2024-03-23T14:36:07.930076+0000","last_fullsized":"2024-03-23T14:36:07.930076+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:58.219983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.24","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302226+0000","last_change":"2024-03-23T14:36:06.949851+0000","last_active":"2024-03-23T14:36:08.302226+0000","last_peered":"2024-03-23T14:36:08.302226+0000","last_clean":"2024-03-23T14:36:08.302226+0000","last_became_active":"2024-03-23T14:36:06.949459+0000","last_became_peered":"2024-03-23T14:36:06.949459+0000","last_unstale":"2024-03-23T14:36:08.302226+0000","last_undegraded":"2024-03-23T14:36:08.302226+0000","last_fullsized":"2024-03-23T14:36:08.302226+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:00:01.725011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.23","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930661+0000","last_change":"2024-03-23T14:36:06.941217+0000","last_active":"2024-03-23T14:36:07.930661+0000","last_peered":"2024-03-23T14:36:07.930661+0000","last_clean":"2024-03-23T14:36:07.930661+0000","last_became_active":"2024-03-23T14:36:06.940936+0000","last_became_peered":"2024-03-23T14:36:06.940936+0000","last_unstale":"2024-03-23T14:36:07.930661+0000","last_undegraded":"2024-03-23T14:36:07.930661+0000","last_fullsized":"2024-03-23T14:36:07.930661+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:37:32.710021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.22","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195265+0000","last_change":"2024-03-23T14:36:06.949519+0000","last_active":"2024-03-23T14:36:08.195265+0000","last_peered":"2024-03-23T14:36:08.195265+0000","last_clean":"2024-03-23T14:36:08.195265+0000","last_became_active":"2024-03-23T14:36:06.949262+0000","last_became_peered":"2024-03-23T14:36:06.949262+0000","last_unstale":"2024-03-23T14:36:08.195265+0000","last_undegraded":"2024-03-23T14:36:08.195265+0000","last_fullsized":"2024-03-23T14:36:08.195265+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:29:22.028913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.21","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301761+0000","last_change":"2024-03-23T14:36:06.948957+0000","last_active":"2024-03-23T14:36:08.301761+0000","last_peered":"2024-03-23T14:36:08.301761+0000","last_clean":"2024-03-23T14:36:08.301761+0000","last_became_active":"2024-03-23T14:36:06.948241+0000","last_became_peered":"2024-03-23T14:36:06.948241+0000","last_unstale":"2024-03-23T14:36:08.301761+0000","last_undegraded":"2024-03-23T14:36:08.301761+0000","last_fullsized":"2024-03-23T14:36:08.301761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:07:29.475104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.20","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195440+0000","last_change":"2024-03-23T14:36:06.951872+0000","last_active":"2024-03-23T14:36:08.195440+0000","last_peered":"2024-03-23T14:36:08.195440+0000","last_clean":"2024-03-23T14:36:08.195440+0000","last_became_active":"2024-03-23T14:36:06.950974+0000","last_became_peered":"2024-03-23T14:36:06.950974+0000","last_unstale":"2024-03-23T14:36:08.195440+0000","last_undegraded":"2024-03-23T14:36:08.195440+0000","last_fullsized":"2024-03-23T14:36:08.195440+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:00:01.725011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.928922+0000","last_change":"2024-03-23T14:36:06.940228+0000","last_active":"2024-03-23T14:36:07.928922+0000","last_peered":"2024-03-23T14:36:07.928922+0000","last_clean":"2024-03-23T14:36:07.928922+0000","last_became_active":"2024-03-23T14:36:06.939927+0000","last_became_peered":"2024-03-23T14:36:06.939927+0000","last_unstale":"2024-03-23T14:36:07.928922+0000","last_undegraded":"2024-03-23T14:36:07.928922+0000","last_fullsized":"2024-03-23T14:36:07.928922+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:14:30.764439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"32'4","reported_seq":18,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302251+0000","last_change":"2024-03-23T14:36:05.938929+0000","last_active":"2024-03-23T14:36:08.302251+0000","last_peered":"2024-03-23T14:36:08.302251+0000","last_clean":"2024-03-23T14:36:08.302251+0000","last_became_active":"2024-03-23T14:36:05.938601+0000","last_became_peered":"2024-03-23T14:36:05.938601+0000","last_unstale":"2024-03-23T14:36:08.302251+0000","last_undegraded":"2024-03-23T14:36:08.302251+0000","last_fullsized":"2024-03-23T14:36:08.302251+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:25:10.296278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302248+0000","last_change":"2024-03-23T14:36:06.950249+0000","last_active":"2024-03-23T14:36:08.302248+0000","last_peered":"2024-03-23T14:36:08.302248+0000","last_clean":"2024-03-23T14:36:08.302248+0000","last_became_active":"2024-03-23T14:36:06.949993+0000","last_became_peered":"2024-03-23T14:36:06.949993+0000","last_unstale":"2024-03-23T14:36:08.302248+0000","last_undegraded":"2024-03-23T14:36:08.302248+0000","last_fullsized":"2024-03-23T14:36:08.302248+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:37.293562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1e","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.928874+0000","last_change":"2024-03-23T14:36:05.936153+0000","last_active":"2024-03-23T14:36:07.928874+0000","last_peered":"2024-03-23T14:36:07.928874+0000","last_clean":"2024-03-23T14:36:07.928874+0000","last_became_active":"2024-03-23T14:36:05.935839+0000","last_became_peered":"2024-03-23T14:36:05.935839+0000","last_unstale":"2024-03-23T14:36:07.928874+0000","last_undegraded":"2024-03-23T14:36:07.928874+0000","last_fullsized":"2024-03-23T14:36:07.928874+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:43:48.936575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930595+0000","last_change":"2024-03-23T14:36:06.942677+0000","last_active":"2024-03-23T14:36:07.930595+0000","last_peered":"2024-03-23T14:36:07.930595+0000","last_clean":"2024-03-23T14:36:07.930595+0000","last_became_active":"2024-03-23T14:36:06.938839+0000","last_became_peered":"2024-03-23T14:36:06.938839+0000","last_unstale":"2024-03-23T14:36:07.930595+0000","last_undegraded":"2024-03-23T14:36:07.930595+0000","last_fullsized":"2024-03-23T14:36:07.930595+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:05:35.559868+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302848+0000","last_change":"2024-03-23T14:36:05.934888+0000","last_active":"2024-03-23T14:36:08.302848+0000","last_peered":"2024-03-23T14:36:08.302848+0000","last_clean":"2024-03-23T14:36:08.302848+0000","last_became_active":"2024-03-23T14:36:05.934620+0000","last_became_peered":"2024-03-23T14:36:05.934620+0000","last_unstale":"2024-03-23T14:36:08.302848+0000","last_undegraded":"2024-03-23T14:36:08.302848+0000","last_fullsized":"2024-03-23T14:36:08.302848+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:08:32.612933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930123+0000","last_change":"2024-03-23T14:36:06.951275+0000","last_active":"2024-03-23T14:36:07.930123+0000","last_peered":"2024-03-23T14:36:07.930123+0000","last_clean":"2024-03-23T14:36:07.930123+0000","last_became_active":"2024-03-23T14:36:06.950497+0000","last_became_peered":"2024-03-23T14:36:06.950497+0000","last_unstale":"2024-03-23T14:36:07.930123+0000","last_undegraded":"2024-03-23T14:36:07.930123+0000","last_fullsized":"2024-03-23T14:36:07.930123+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:02:20.025127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302401+0000","last_change":"2024-03-23T14:36:05.939290+0000","last_active":"2024-03-23T14:36:08.302401+0000","last_peered":"2024-03-23T14:36:08.302401+0000","last_clean":"2024-03-23T14:36:08.302401+0000","last_became_active":"2024-03-23T14:36:05.938947+0000","last_became_peered":"2024-03-23T14:36:05.938947+0000","last_unstale":"2024-03-23T14:36:08.302401+0000","last_undegraded":"2024-03-23T14:36:08.302401+0000","last_fullsized":"2024-03-23T14:36:08.302401+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:59:51.379843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195409+0000","last_change":"2024-03-23T14:36:06.951964+0000","last_active":"2024-03-23T14:36:08.195409+0000","last_peered":"2024-03-23T14:36:08.195409+0000","last_clean":"2024-03-23T14:36:08.195409+0000","last_became_active":"2024-03-23T14:36:06.951621+0000","last_became_peered":"2024-03-23T14:36:06.951621+0000","last_unstale":"2024-03-23T14:36:08.195409+0000","last_undegraded":"2024-03-23T14:36:08.195409+0000","last_fullsized":"2024-03-23T14:36:08.195409+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:14:50.221562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1b","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930612+0000","last_change":"2024-03-23T14:36:05.934708+0000","last_active":"2024-03-23T14:36:07.930612+0000","last_peered":"2024-03-23T14:36:07.930612+0000","last_clean":"2024-03-23T14:36:07.930612+0000","last_became_active":"2024-03-23T14:36:05.934390+0000","last_became_peered":"2024-03-23T14:36:05.934390+0000","last_unstale":"2024-03-23T14:36:07.930612+0000","last_undegraded":"2024-03-23T14:36:07.930612+0000","last_fullsized":"2024-03-23T14:36:07.930612+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:11:59.139317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301933+0000","last_change":"2024-03-23T14:36:06.946215+0000","last_active":"2024-03-23T14:36:08.301933+0000","last_peered":"2024-03-23T14:36:08.301933+0000","last_clean":"2024-03-23T14:36:08.301933+0000","last_became_active":"2024-03-23T14:36:06.945877+0000","last_became_peered":"2024-03-23T14:36:06.945877+0000","last_unstale":"2024-03-23T14:36:08.301933+0000","last_undegraded":"2024-03-23T14:36:08.301933+0000","last_fullsized":"2024-03-23T14:36:08.301933+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:02:20.025127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930379+0000","last_change":"2024-03-23T14:36:05.934232+0000","last_active":"2024-03-23T14:36:07.930379+0000","last_peered":"2024-03-23T14:36:07.930379+0000","last_clean":"2024-03-23T14:36:07.930379+0000","last_became_active":"2024-03-23T14:36:05.933938+0000","last_became_peered":"2024-03-23T14:36:05.933938+0000","last_unstale":"2024-03-23T14:36:07.930379+0000","last_undegraded":"2024-03-23T14:36:07.930379+0000","last_fullsized":"2024-03-23T14:36:07.930379+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:08:47.902391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930355+0000","last_change":"2024-03-23T14:36:06.946014+0000","last_active":"2024-03-23T14:36:07.930355+0000","last_peered":"2024-03-23T14:36:07.930355+0000","last_clean":"2024-03-23T14:36:07.930355+0000","last_became_active":"2024-03-23T14:36:06.945733+0000","last_became_peered":"2024-03-23T14:36:06.945733+0000","last_unstale":"2024-03-23T14:36:07.930355+0000","last_undegraded":"2024-03-23T14:36:07.930355+0000","last_fullsized":"2024-03-23T14:36:07.930355+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:33:20.427109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302511+0000","last_change":"2024-03-23T14:36:05.938825+0000","last_active":"2024-03-23T14:36:08.302511+0000","last_peered":"2024-03-23T14:36:08.302511+0000","last_clean":"2024-03-23T14:36:08.302511+0000","last_became_active":"2024-03-23T14:36:05.938569+0000","last_became_peered":"2024-03-23T14:36:05.938569+0000","last_unstale":"2024-03-23T14:36:08.302511+0000","last_undegraded":"2024-03-23T14:36:08.302511+0000","last_fullsized":"2024-03-23T14:36:08.302511+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:12.171389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929477+0000","last_change":"2024-03-23T14:36:06.942209+0000","last_active":"2024-03-23T14:36:07.929477+0000","last_peered":"2024-03-23T14:36:07.929477+0000","last_clean":"2024-03-23T14:36:07.929477+0000","last_became_active":"2024-03-23T14:36:06.941926+0000","last_became_peered":"2024-03-23T14:36:06.941926+0000","last_unstale":"2024-03-23T14:36:07.929477+0000","last_undegraded":"2024-03-23T14:36:07.929477+0000","last_fullsized":"2024-03-23T14:36:07.929477+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:33:20.427109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929944+0000","last_change":"2024-03-23T14:36:05.930166+0000","last_active":"2024-03-23T14:36:07.929944+0000","last_peered":"2024-03-23T14:36:07.929944+0000","last_clean":"2024-03-23T14:36:07.929944+0000","last_became_active":"2024-03-23T14:36:05.929607+0000","last_became_peered":"2024-03-23T14:36:05.929607+0000","last_unstale":"2024-03-23T14:36:07.929944+0000","last_undegraded":"2024-03-23T14:36:07.929944+0000","last_fullsized":"2024-03-23T14:36:07.929944+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:19:54.216869+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302398+0000","last_change":"2024-03-23T14:36:06.944279+0000","last_active":"2024-03-23T14:36:08.302398+0000","last_peered":"2024-03-23T14:36:08.302398+0000","last_clean":"2024-03-23T14:36:08.302398+0000","last_became_active":"2024-03-23T14:36:06.943894+0000","last_became_peered":"2024-03-23T14:36:06.943894+0000","last_unstale":"2024-03-23T14:36:08.302398+0000","last_undegraded":"2024-03-23T14:36:08.302398+0000","last_fullsized":"2024-03-23T14:36:08.302398+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:43:54.788598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.303059+0000","last_change":"2024-03-23T14:36:05.936008+0000","last_active":"2024-03-23T14:36:08.303059+0000","last_peered":"2024-03-23T14:36:08.303059+0000","last_clean":"2024-03-23T14:36:08.303059+0000","last_became_active":"2024-03-23T14:36:05.935693+0000","last_became_peered":"2024-03-23T14:36:05.935693+0000","last_unstale":"2024-03-23T14:36:08.303059+0000","last_undegraded":"2024-03-23T14:36:08.303059+0000","last_fullsized":"2024-03-23T14:36:08.303059+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:23:51.989867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.303082+0000","last_change":"2024-03-23T14:36:06.938877+0000","last_active":"2024-03-23T14:36:08.303082+0000","last_peered":"2024-03-23T14:36:08.303082+0000","last_clean":"2024-03-23T14:36:08.303082+0000","last_became_active":"2024-03-23T14:36:06.938578+0000","last_became_peered":"2024-03-23T14:36:06.938578+0000","last_unstale":"2024-03-23T14:36:08.303082+0000","last_undegraded":"2024-03-23T14:36:08.303082+0000","last_fullsized":"2024-03-23T14:36:08.303082+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:27:44.094690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929091+0000","last_change":"2024-03-23T14:36:05.928987+0000","last_active":"2024-03-23T14:36:07.929091+0000","last_peered":"2024-03-23T14:36:07.929091+0000","last_clean":"2024-03-23T14:36:07.929091+0000","last_became_active":"2024-03-23T14:36:05.928750+0000","last_became_peered":"2024-03-23T14:36:05.928750+0000","last_unstale":"2024-03-23T14:36:07.929091+0000","last_undegraded":"2024-03-23T14:36:07.929091+0000","last_fullsized":"2024-03-23T14:36:07.929091+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:28.610944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302910+0000","last_change":"2024-03-23T14:36:06.941560+0000","last_active":"2024-03-23T14:36:08.302910+0000","last_peered":"2024-03-23T14:36:08.302910+0000","last_clean":"2024-03-23T14:36:08.302910+0000","last_became_active":"2024-03-23T14:36:06.941311+0000","last_became_peered":"2024-03-23T14:36:06.941311+0000","last_unstale":"2024-03-23T14:36:08.302910+0000","last_undegraded":"2024-03-23T14:36:08.302910+0000","last_fullsized":"2024-03-23T14:36:08.302910+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:43:54.788598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.15","version":"32'3","reported_seq":17,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302162+0000","last_change":"2024-03-23T14:36:05.939208+0000","last_active":"2024-03-23T14:36:08.302162+0000","last_peered":"2024-03-23T14:36:08.302162+0000","last_clean":"2024-03-23T14:36:08.302162+0000","last_became_active":"2024-03-23T14:36:05.938925+0000","last_became_peered":"2024-03-23T14:36:05.938925+0000","last_unstale":"2024-03-23T14:36:08.302162+0000","last_undegraded":"2024-03-23T14:36:08.302162+0000","last_fullsized":"2024-03-23T14:36:08.302162+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:08:32.612933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.15","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301904+0000","last_change":"2024-03-23T14:36:06.955753+0000","last_active":"2024-03-23T14:36:08.301904+0000","last_peered":"2024-03-23T14:36:08.301904+0000","last_clean":"2024-03-23T14:36:08.301904+0000","last_became_active":"2024-03-23T14:36:06.955449+0000","last_became_peered":"2024-03-23T14:36:06.955449+0000","last_unstale":"2024-03-23T14:36:08.301904+0000","last_undegraded":"2024-03-23T14:36:08.301904+0000","last_fullsized":"2024-03-23T14:36:08.301904+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:58.219983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.14","version":"32'6","reported_seq":20,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302867+0000","last_change":"2024-03-23T14:36:05.934454+0000","last_active":"2024-03-23T14:36:08.302867+0000","last_peered":"2024-03-23T14:36:08.302867+0000","last_clean":"2024-03-23T14:36:08.302867+0000","last_became_active":"2024-03-23T14:36:05.934204+0000","last_became_peered":"2024-03-23T14:36:05.934204+0000","last_unstale":"2024-03-23T14:36:08.302867+0000","last_undegraded":"2024-03-23T14:36:08.302867+0000","last_fullsized":"2024-03-23T14:36:08.302867+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:31:53.393673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929539+0000","last_change":"2024-03-23T14:36:06.942416+0000","last_active":"2024-03-23T14:36:07.929539+0000","last_peered":"2024-03-23T14:36:07.929539+0000","last_clean":"2024-03-23T14:36:07.929539+0000","last_became_active":"2024-03-23T14:36:06.942068+0000","last_became_peered":"2024-03-23T14:36:06.942068+0000","last_unstale":"2024-03-23T14:36:07.929539+0000","last_undegraded":"2024-03-23T14:36:07.929539+0000","last_fullsized":"2024-03-23T14:36:07.929539+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:48:52.462180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.13","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930718+0000","last_change":"2024-03-23T14:36:05.929565+0000","last_active":"2024-03-23T14:36:07.930718+0000","last_peered":"2024-03-23T14:36:07.930718+0000","last_clean":"2024-03-23T14:36:07.930718+0000","last_became_active":"2024-03-23T14:36:05.929288+0000","last_became_peered":"2024-03-23T14:36:05.929288+0000","last_unstale":"2024-03-23T14:36:07.930718+0000","last_undegraded":"2024-03-23T14:36:07.930718+0000","last_fullsized":"2024-03-23T14:36:07.930718+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:25:10.296278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302860+0000","last_change":"2024-03-23T14:36:06.940544+0000","last_active":"2024-03-23T14:36:08.302860+0000","last_peered":"2024-03-23T14:36:08.302860+0000","last_clean":"2024-03-23T14:36:08.302860+0000","last_became_active":"2024-03-23T14:36:06.940250+0000","last_became_peered":"2024-03-23T14:36:06.940250+0000","last_unstale":"2024-03-23T14:36:08.302860+0000","last_undegraded":"2024-03-23T14:36:08.302860+0000","last_fullsized":"2024-03-23T14:36:08.302860+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:00:01.725011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929470+0000","last_change":"2024-03-23T14:36:05.931244+0000","last_active":"2024-03-23T14:36:07.929470+0000","last_peered":"2024-03-23T14:36:07.929470+0000","last_clean":"2024-03-23T14:36:07.929470+0000","last_became_active":"2024-03-23T14:36:05.930943+0000","last_became_peered":"2024-03-23T14:36:05.930943+0000","last_unstale":"2024-03-23T14:36:07.929470+0000","last_undegraded":"2024-03-23T14:36:07.929470+0000","last_fullsized":"2024-03-23T14:36:07.929470+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:28.610944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195385+0000","last_change":"2024-03-23T14:36:06.950962+0000","last_active":"2024-03-23T14:36:08.195385+0000","last_peered":"2024-03-23T14:36:08.195385+0000","last_clean":"2024-03-23T14:36:08.195385+0000","last_became_active":"2024-03-23T14:36:06.950312+0000","last_became_peered":"2024-03-23T14:36:06.950312+0000","last_unstale":"2024-03-23T14:36:08.195385+0000","last_undegraded":"2024-03-23T14:36:08.195385+0000","last_fullsized":"2024-03-23T14:36:08.195385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:02:20.025127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302509+0000","last_change":"2024-03-23T14:36:05.938614+0000","last_active":"2024-03-23T14:36:08.302509+0000","last_peered":"2024-03-23T14:36:08.302509+0000","last_clean":"2024-03-23T14:36:08.302509+0000","last_became_active":"2024-03-23T14:36:05.938355+0000","last_became_peered":"2024-03-23T14:36:05.938355+0000","last_unstale":"2024-03-23T14:36:08.302509+0000","last_undegraded":"2024-03-23T14:36:08.302509+0000","last_fullsized":"2024-03-23T14:36:08.302509+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:23.403007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301813+0000","last_change":"2024-03-23T14:36:06.950204+0000","last_active":"2024-03-23T14:36:08.301813+0000","last_peered":"2024-03-23T14:36:08.301813+0000","last_clean":"2024-03-23T14:36:08.301813+0000","last_became_active":"2024-03-23T14:36:06.949937+0000","last_became_peered":"2024-03-23T14:36:06.949937+0000","last_unstale":"2024-03-23T14:36:08.301813+0000","last_undegraded":"2024-03-23T14:36:08.301813+0000","last_fullsized":"2024-03-23T14:36:08.301813+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:15:20.453546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"32'8","reported_seq":22,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302315+0000","last_change":"2024-03-23T14:36:05.935233+0000","last_active":"2024-03-23T14:36:08.302315+0000","last_peered":"2024-03-23T14:36:08.302315+0000","last_clean":"2024-03-23T14:36:08.302315+0000","last_became_active":"2024-03-23T14:36:05.934962+0000","last_became_peered":"2024-03-23T14:36:05.934962+0000","last_unstale":"2024-03-23T14:36:08.302315+0000","last_undegraded":"2024-03-23T14:36:08.302315+0000","last_fullsized":"2024-03-23T14:36:08.302315+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:07:28.471125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930484+0000","last_change":"2024-03-23T14:36:06.942627+0000","last_active":"2024-03-23T14:36:07.930484+0000","last_peered":"2024-03-23T14:36:07.930484+0000","last_clean":"2024-03-23T14:36:07.930484+0000","last_became_active":"2024-03-23T14:36:06.942304+0000","last_became_peered":"2024-03-23T14:36:06.942304+0000","last_unstale":"2024-03-23T14:36:07.930484+0000","last_undegraded":"2024-03-23T14:36:07.930484+0000","last_fullsized":"2024-03-23T14:36:07.930484+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:48:52.462180+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.f","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929905+0000","last_change":"2024-03-23T14:36:05.930020+0000","last_active":"2024-03-23T14:36:07.929905+0000","last_peered":"2024-03-23T14:36:07.929905+0000","last_clean":"2024-03-23T14:36:07.929905+0000","last_became_active":"2024-03-23T14:36:05.929756+0000","last_became_peered":"2024-03-23T14:36:05.929756+0000","last_unstale":"2024-03-23T14:36:07.929905+0000","last_undegraded":"2024-03-23T14:36:07.929905+0000","last_fullsized":"2024-03-23T14:36:07.929905+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:11:59.139317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302420+0000","last_change":"2024-03-23T14:36:06.952467+0000","last_active":"2024-03-23T14:36:08.302420+0000","last_peered":"2024-03-23T14:36:08.302420+0000","last_clean":"2024-03-23T14:36:08.302420+0000","last_became_active":"2024-03-23T14:36:06.952172+0000","last_became_peered":"2024-03-23T14:36:06.952172+0000","last_unstale":"2024-03-23T14:36:08.302420+0000","last_undegraded":"2024-03-23T14:36:08.302420+0000","last_fullsized":"2024-03-23T14:36:08.302420+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:39:21.302001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930469+0000","last_change":"2024-03-23T14:36:05.929216+0000","last_active":"2024-03-23T14:36:07.930469+0000","last_peered":"2024-03-23T14:36:07.930469+0000","last_clean":"2024-03-23T14:36:07.930469+0000","last_became_active":"2024-03-23T14:36:05.928955+0000","last_became_peered":"2024-03-23T14:36:05.928955+0000","last_unstale":"2024-03-23T14:36:07.930469+0000","last_undegraded":"2024-03-23T14:36:07.930469+0000","last_fullsized":"2024-03-23T14:36:07.930469+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:00:35.991217+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929807+0000","last_change":"2024-03-23T14:36:06.951795+0000","last_active":"2024-03-23T14:36:07.929807+0000","last_peered":"2024-03-23T14:36:07.929807+0000","last_clean":"2024-03-23T14:36:07.929807+0000","last_became_active":"2024-03-23T14:36:06.951164+0000","last_became_peered":"2024-03-23T14:36:06.951164+0000","last_unstale":"2024-03-23T14:36:07.929807+0000","last_undegraded":"2024-03-23T14:36:07.929807+0000","last_fullsized":"2024-03-23T14:36:07.929807+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:37:32.710021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.d","version":"32'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930430+0000","last_change":"2024-03-23T14:36:05.938973+0000","last_active":"2024-03-23T14:36:07.930430+0000","last_peered":"2024-03-23T14:36:07.930430+0000","last_clean":"2024-03-23T14:36:07.930430+0000","last_became_active":"2024-03-23T14:36:05.938700+0000","last_became_peered":"2024-03-23T14:36:05.938700+0000","last_unstale":"2024-03-23T14:36:07.930430+0000","last_undegraded":"2024-03-23T14:36:07.930430+0000","last_fullsized":"2024-03-23T14:36:07.930430+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:02:37.262185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302457+0000","last_change":"2024-03-23T14:36:06.939696+0000","last_active":"2024-03-23T14:36:08.302457+0000","last_peered":"2024-03-23T14:36:08.302457+0000","last_clean":"2024-03-23T14:36:08.302457+0000","last_became_active":"2024-03-23T14:36:06.939438+0000","last_became_peered":"2024-03-23T14:36:06.939438+0000","last_unstale":"2024-03-23T14:36:08.302457+0000","last_undegraded":"2024-03-23T14:36:08.302457+0000","last_fullsized":"2024-03-23T14:36:08.302457+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:25:11.300257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930361+0000","last_change":"2024-03-23T14:36:05.927882+0000","last_active":"2024-03-23T14:36:07.930361+0000","last_peered":"2024-03-23T14:36:07.930361+0000","last_clean":"2024-03-23T14:36:07.930361+0000","last_became_active":"2024-03-23T14:36:05.927594+0000","last_became_peered":"2024-03-23T14:36:05.927594+0000","last_unstale":"2024-03-23T14:36:07.930361+0000","last_undegraded":"2024-03-23T14:36:07.930361+0000","last_fullsized":"2024-03-23T14:36:07.930361+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:32:34.335118+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302344+0000","last_change":"2024-03-23T14:36:06.949467+0000","last_active":"2024-03-23T14:36:08.302344+0000","last_peered":"2024-03-23T14:36:08.302344+0000","last_clean":"2024-03-23T14:36:08.302344+0000","last_became_active":"2024-03-23T14:36:06.949058+0000","last_became_peered":"2024-03-23T14:36:06.949058+0000","last_unstale":"2024-03-23T14:36:08.302344+0000","last_undegraded":"2024-03-23T14:36:08.302344+0000","last_fullsized":"2024-03-23T14:36:08.302344+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:43:49.940554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.b","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.928957+0000","last_change":"2024-03-23T14:36:05.930251+0000","last_active":"2024-03-23T14:36:07.928957+0000","last_peered":"2024-03-23T14:36:07.928957+0000","last_clean":"2024-03-23T14:36:07.928957+0000","last_became_active":"2024-03-23T14:36:05.930015+0000","last_became_peered":"2024-03-23T14:36:05.930015+0000","last_unstale":"2024-03-23T14:36:07.928957+0000","last_undegraded":"2024-03-23T14:36:07.928957+0000","last_fullsized":"2024-03-23T14:36:07.928957+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:11:59.139317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302401+0000","last_change":"2024-03-23T14:36:06.946259+0000","last_active":"2024-03-23T14:36:08.302401+0000","last_peered":"2024-03-23T14:36:08.302401+0000","last_clean":"2024-03-23T14:36:08.302401+0000","last_became_active":"2024-03-23T14:36:06.945873+0000","last_became_peered":"2024-03-23T14:36:06.945873+0000","last_unstale":"2024-03-23T14:36:08.302401+0000","last_undegraded":"2024-03-23T14:36:08.302401+0000","last_fullsized":"2024-03-23T14:36:08.302401+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:12:00.143296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301684+0000","last_change":"2024-03-23T14:36:05.934694+0000","last_active":"2024-03-23T14:36:08.301684+0000","last_peered":"2024-03-23T14:36:08.301684+0000","last_clean":"2024-03-23T14:36:08.301684+0000","last_became_active":"2024-03-23T14:36:05.934376+0000","last_became_peered":"2024-03-23T14:36:05.934376+0000","last_unstale":"2024-03-23T14:36:08.301684+0000","last_undegraded":"2024-03-23T14:36:08.301684+0000","last_fullsized":"2024-03-23T14:36:08.301684+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:11:59.139317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302251+0000","last_change":"2024-03-23T14:36:06.940704+0000","last_active":"2024-03-23T14:36:08.302251+0000","last_peered":"2024-03-23T14:36:08.302251+0000","last_clean":"2024-03-23T14:36:08.302251+0000","last_became_active":"2024-03-23T14:36:06.940429+0000","last_became_peered":"2024-03-23T14:36:06.940429+0000","last_unstale":"2024-03-23T14:36:08.302251+0000","last_undegraded":"2024-03-23T14:36:08.302251+0000","last_fullsized":"2024-03-23T14:36:08.302251+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:32:35.339097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929999+0000","last_change":"2024-03-23T14:36:05.931786+0000","last_active":"2024-03-23T14:36:07.929999+0000","last_peered":"2024-03-23T14:36:07.929999+0000","last_clean":"2024-03-23T14:36:07.929999+0000","last_became_active":"2024-03-23T14:36:05.931494+0000","last_became_peered":"2024-03-23T14:36:05.931494+0000","last_unstale":"2024-03-23T14:36:07.929999+0000","last_undegraded":"2024-03-23T14:36:07.929999+0000","last_fullsized":"2024-03-23T14:36:07.929999+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:43:48.936575+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195242+0000","last_change":"2024-03-23T14:36:06.947506+0000","last_active":"2024-03-23T14:36:08.195242+0000","last_peered":"2024-03-23T14:36:08.195242+0000","last_clean":"2024-03-23T14:36:08.195242+0000","last_became_active":"2024-03-23T14:36:06.947215+0000","last_became_peered":"2024-03-23T14:36:06.947215+0000","last_unstale":"2024-03-23T14:36:08.195242+0000","last_undegraded":"2024-03-23T14:36:08.195242+0000","last_fullsized":"2024-03-23T14:36:08.195242+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:37:32.710021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301813+0000","last_change":"2024-03-23T14:36:05.938759+0000","last_active":"2024-03-23T14:36:08.301813+0000","last_peered":"2024-03-23T14:36:08.301813+0000","last_clean":"2024-03-23T14:36:08.301813+0000","last_became_active":"2024-03-23T14:36:05.938441+0000","last_became_peered":"2024-03-23T14:36:05.938441+0000","last_unstale":"2024-03-23T14:36:08.301813+0000","last_undegraded":"2024-03-23T14:36:08.301813+0000","last_fullsized":"2024-03-23T14:36:08.301813+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:31:53.393673+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302464+0000","last_change":"2024-03-23T14:36:06.943045+0000","last_active":"2024-03-23T14:36:08.302464+0000","last_peered":"2024-03-23T14:36:08.302464+0000","last_clean":"2024-03-23T14:36:08.302464+0000","last_became_active":"2024-03-23T14:36:06.942004+0000","last_became_peered":"2024-03-23T14:36:06.942004+0000","last_unstale":"2024-03-23T14:36:08.302464+0000","last_undegraded":"2024-03-23T14:36:08.302464+0000","last_fullsized":"2024-03-23T14:36:08.302464+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:29.614924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"32'2","reported_seq":16,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929926+0000","last_change":"2024-03-23T14:36:05.939881+0000","last_active":"2024-03-23T14:36:07.929926+0000","last_peered":"2024-03-23T14:36:07.929926+0000","last_clean":"2024-03-23T14:36:07.929926+0000","last_became_active":"2024-03-23T14:36:05.939607+0000","last_became_peered":"2024-03-23T14:36:05.939607+0000","last_unstale":"2024-03-23T14:36:07.929926+0000","last_undegraded":"2024-03-23T14:36:07.929926+0000","last_fullsized":"2024-03-23T14:36:07.929926+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:25:10.296278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302467+0000","last_change":"2024-03-23T14:36:06.953858+0000","last_active":"2024-03-23T14:36:08.302467+0000","last_peered":"2024-03-23T14:36:08.302467+0000","last_clean":"2024-03-23T14:36:08.302467+0000","last_became_active":"2024-03-23T14:36:06.953593+0000","last_became_peered":"2024-03-23T14:36:06.953593+0000","last_unstale":"2024-03-23T14:36:08.302467+0000","last_undegraded":"2024-03-23T14:36:08.302467+0000","last_fullsized":"2024-03-23T14:36:08.302467+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:54:12.738117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.6","version":"33'1","reported_seq":15,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929045+0000","last_change":"2024-03-23T14:36:05.930703+0000","last_active":"2024-03-23T14:36:07.929045+0000","last_peered":"2024-03-23T14:36:07.929045+0000","last_clean":"2024-03-23T14:36:07.929045+0000","last_became_active":"2024-03-23T14:36:05.930418+0000","last_became_peered":"2024-03-23T14:36:05.930418+0000","last_unstale":"2024-03-23T14:36:07.929045+0000","last_undegraded":"2024-03-23T14:36:07.929045+0000","last_fullsized":"2024-03-23T14:36:07.929045+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:08:47.902391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.928745+0000","last_change":"2024-03-23T14:36:06.944491+0000","last_active":"2024-03-23T14:36:07.928745+0000","last_peered":"2024-03-23T14:36:07.928745+0000","last_clean":"2024-03-23T14:36:07.928745+0000","last_became_active":"2024-03-23T14:36:06.944226+0000","last_became_peered":"2024-03-23T14:36:06.944226+0000","last_unstale":"2024-03-23T14:36:07.928745+0000","last_undegraded":"2024-03-23T14:36:07.928745+0000","last_fullsized":"2024-03-23T14:36:07.928745+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:07:29.475104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929796+0000","last_change":"2024-03-23T14:36:05.933556+0000","last_active":"2024-03-23T14:36:07.929796+0000","last_peered":"2024-03-23T14:36:07.929796+0000","last_clean":"2024-03-23T14:36:07.929796+0000","last_became_active":"2024-03-23T14:36:05.933289+0000","last_became_peered":"2024-03-23T14:36:05.933289+0000","last_unstale":"2024-03-23T14:36:07.929796+0000","last_undegraded":"2024-03-23T14:36:07.929796+0000","last_fullsized":"2024-03-23T14:36:07.929796+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:24:23.403007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.301761+0000","last_change":"2024-03-23T14:36:06.949483+0000","last_active":"2024-03-23T14:36:08.301761+0000","last_peered":"2024-03-23T14:36:08.301761+0000","last_clean":"2024-03-23T14:36:08.301761+0000","last_became_active":"2024-03-23T14:36:06.948977+0000","last_became_peered":"2024-03-23T14:36:06.948977+0000","last_unstale":"2024-03-23T14:36:08.301761+0000","last_undegraded":"2024-03-23T14:36:08.301761+0000","last_fullsized":"2024-03-23T14:36:08.301761+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:23:52.993846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929060+0000","last_change":"2024-03-23T14:36:05.933333+0000","last_active":"2024-03-23T14:36:07.929060+0000","last_peered":"2024-03-23T14:36:07.929060+0000","last_clean":"2024-03-23T14:36:07.929060+0000","last_became_active":"2024-03-23T14:36:05.933061+0000","last_became_peered":"2024-03-23T14:36:05.933061+0000","last_unstale":"2024-03-23T14:36:07.929060+0000","last_undegraded":"2024-03-23T14:36:07.929060+0000","last_fullsized":"2024-03-23T14:36:07.929060+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:25:10.296278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.3","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.928908+0000","last_change":"2024-03-23T14:36:06.946410+0000","last_active":"2024-03-23T14:36:07.928908+0000","last_peered":"2024-03-23T14:36:07.928908+0000","last_clean":"2024-03-23T14:36:07.928908+0000","last_became_active":"2024-03-23T14:36:06.946072+0000","last_became_peered":"2024-03-23T14:36:06.946072+0000","last_unstale":"2024-03-23T14:36:07.928908+0000","last_undegraded":"2024-03-23T14:36:07.928908+0000","last_fullsized":"2024-03-23T14:36:07.928908+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:15:20.453546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.929561+0000","last_change":"2024-03-23T14:36:05.933397+0000","last_active":"2024-03-23T14:36:07.929561+0000","last_peered":"2024-03-23T14:36:07.929561+0000","last_clean":"2024-03-23T14:36:07.929561+0000","last_became_active":"2024-03-23T14:36:05.933091+0000","last_became_peered":"2024-03-23T14:36:05.933091+0000","last_unstale":"2024-03-23T14:36:07.929561+0000","last_undegraded":"2024-03-23T14:36:07.929561+0000","last_fullsized":"2024-03-23T14:36:07.929561+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:39:12.171389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302433+0000","last_change":"2024-03-23T14:36:06.943816+0000","last_active":"2024-03-23T14:36:08.302433+0000","last_peered":"2024-03-23T14:36:08.302433+0000","last_clean":"2024-03-23T14:36:08.302433+0000","last_became_active":"2024-03-23T14:36:06.942998+0000","last_became_peered":"2024-03-23T14:36:06.942998+0000","last_unstale":"2024-03-23T14:36:08.302433+0000","last_undegraded":"2024-03-23T14:36:08.302433+0000","last_fullsized":"2024-03-23T14:36:08.302433+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:02:20.025127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930087+0000","last_change":"2024-03-23T14:36:05.939560+0000","last_active":"2024-03-23T14:36:07.930087+0000","last_peered":"2024-03-23T14:36:07.930087+0000","last_clean":"2024-03-23T14:36:07.930087+0000","last_became_active":"2024-03-23T14:36:05.939244+0000","last_became_peered":"2024-03-23T14:36:05.939244+0000","last_unstale":"2024-03-23T14:36:07.930087+0000","last_undegraded":"2024-03-23T14:36:07.930087+0000","last_fullsized":"2024-03-23T14:36:07.930087+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:30:15.923864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:07.930025+0000","last_change":"2024-03-23T14:36:06.935714+0000","last_active":"2024-03-23T14:36:07.930025+0000","last_peered":"2024-03-23T14:36:07.930025+0000","last_clean":"2024-03-23T14:36:07.930025+0000","last_became_active":"2024-03-23T14:36:06.935311+0000","last_became_peered":"2024-03-23T14:36:06.935311+0000","last_unstale":"2024-03-23T14:36:07.930025+0000","last_undegraded":"2024-03-23T14:36:07.930025+0000","last_fullsized":"2024-03-23T14:36:07.930025+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:14:30.764439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302219+0000","last_change":"2024-03-23T14:36:05.936554+0000","last_active":"2024-03-23T14:36:08.302219+0000","last_peered":"2024-03-23T14:36:08.302219+0000","last_clean":"2024-03-23T14:36:08.302219+0000","last_became_active":"2024-03-23T14:36:05.936296+0000","last_became_peered":"2024-03-23T14:36:05.936296+0000","last_unstale":"2024-03-23T14:36:08.302219+0000","last_undegraded":"2024-03-23T14:36:08.302219+0000","last_fullsized":"2024-03-23T14:36:08.302219+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:34:28.610944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":12,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195278+0000","last_change":"2024-03-23T14:36:06.944725+0000","last_active":"2024-03-23T14:36:08.195278+0000","last_peered":"2024-03-23T14:36:08.195278+0000","last_clean":"2024-03-23T14:36:08.195278+0000","last_became_active":"2024-03-23T14:36:06.944478+0000","last_became_peered":"2024-03-23T14:36:06.944478+0000","last_unstale":"2024-03-23T14:36:08.195278+0000","last_undegraded":"2024-03-23T14:36:08.195278+0000","last_fullsized":"2024-03-23T14:36:08.195278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":31,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:05.914757+0000","last_clean_scrub_stamp":"2024-03-23T14:36:05.914757+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:39:58.219983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.302001+0000","last_change":"2024-03-23T14:36:05.933042+0000","last_active":"2024-03-23T14:36:08.302001+0000","last_peered":"2024-03-23T14:36:08.302001+0000","last_clean":"2024-03-23T14:36:08.302001+0000","last_became_active":"2024-03-23T14:36:05.932739+0000","last_became_peered":"2024-03-23T14:36:05.932739+0000","last_unstale":"2024-03-23T14:36:08.302001+0000","last_undegraded":"2024-03-23T14:36:08.302001+0000","last_fullsized":"2024-03-23T14:36:08.302001+0000","mapping_epoch":30,"log_start":"0'0","ondisk_log_start":"0'0","created":30,"last_epoch_clean":31,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:04.910778+0000","last_clean_scrub_stamp":"2024-03-23T14:36:04.910778+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:08:47.902391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":192,"reported_epoch":33,"state":"active+clean","last_fresh":"2024-03-23T14:36:08.195497+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:36:08.195497+0000","last_peered":"2024-03-23T14:36:08.195497+0000","last_clean":"2024-03-23T14:36:08.195497+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:36:08.195497+0000","last_undegraded":"2024-03-23T14:36:08.195497+0000","last_fullsized":"2024-03-23T14:36:08.195497+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":5,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574866,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343868,"statfs":{"total":96636764160,"available":96608120832,"internally_reserved":0,"allocated":970752,"data_stored":737074,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574866,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574865,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":147288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574867,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27396,"kb_used_data":372,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344444,"statfs":{"total":96636764160,"available":96608710656,"internally_reserved":0,"allocated":380928,"data_stored":146720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574865,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574866,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27392,"kb_used_data":368,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344448,"statfs":{"total":96636764160,"available":96608714752,"internally_reserved":0,"allocated":376832,"data_stored":146686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574867,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27404,"kb_used_data":380,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344436,"statfs":{"total":96636764160,"available":96608702464,"internally_reserved":0,"allocated":389120,"data_stored":148250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574866,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343872,"statfs":{"total":96636764160,"available":96608124928,"internally_reserved":0,"allocated":966656,"data_stored":736984,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":70,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:36:15.608 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T14:36:15.608 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T14:36:15.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.755+0000 7f7a8ed2e640 1 Processor -- start 2024-03-23T14:36:15.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.755+0000 7f7a8ed2e640 1 -- start start 2024-03-23T14:36:15.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a8ed2e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7a88160390 0x7f7a880b4880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a8ed2e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a88070d10 con 0x7f7a880d03e0 2024-03-23T14:36:15.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a8ed2e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a88070ff0 con 0x7f7a881454f0 2024-03-23T14:36:15.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a8ed2e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f7a880b4f80 con 0x7f7a88160390 2024-03-23T14:36:15.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a87fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7a880d03e0 0x7f7a880cdf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45470/0 (socket says 172.21.15.161:45470) 2024-03-23T14:36:15.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a87fff640 1 -- 172.21.15.161:0/2679925990 learned_addr learned my addr 172.21.15.161:0/2679925990 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:15.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a8d2cb640 1 --2- 172.21.15.161:0/2679925990 >> v2:172.21.15.183:3300/0 conn(0x7f7a88160390 0x7f7a880b4880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a877fe640 1 -- 172.21.15.161:0/2679925990 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1197690999 0 0) 0x7f7a88070d10 con 0x7f7a880d03e0 2024-03-23T14:36:15.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a877fe640 1 -- 172.21.15.161:0/2679925990 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a78003520 con 0x7f7a880d03e0 2024-03-23T14:36:15.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a8d2cb640 1 -- 172.21.15.161:0/2679925990 >> v1:172.21.15.183:6789/0 conn(0x7f7a881454f0 legacy=0x7f7a880cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a8d2cb640 1 -- 172.21.15.161:0/2679925990 >> v1:172.21.15.161:6789/0 conn(0x7f7a880d03e0 legacy=0x7f7a880cdf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.756+0000 7f7a8d2cb640 1 -- 172.21.15.161:0/2679925990 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a880712d0 con 0x7f7a88160390 2024-03-23T14:36:15.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.757+0000 7f7a8d2cb640 1 --2- 172.21.15.161:0/2679925990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a88160390 0x7f7a880b4880 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009d90 tx=0x7f7a8002beb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b56c5beb645edbdc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:15.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.757+0000 7f7a877fe640 1 -- 172.21.15.161:0/2679925990 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8002cc60 con 0x7f7a88160390 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.757+0000 7f7a877fe640 1 -- 172.21.15.161:0/2679925990 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a8002cdc0 con 0x7f7a88160390 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.757+0000 7f7a877fe640 1 -- 172.21.15.161:0/2679925990 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a80034640 con 0x7f7a88160390 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.757+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/2679925990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a88160390 msgr2=0x7f7a880b4880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.757+0000 7f7a8ed2e640 1 --2- 172.21.15.161:0/2679925990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a88160390 0x7f7a880b4880 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f7a80009d90 tx=0x7f7a8002beb0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.758+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/2679925990 shutdown_connections 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.758+0000 7f7a8ed2e640 1 --2- 172.21.15.161:0/2679925990 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a88160390 0x7f7a880b4880 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.758+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/2679925990 >> 172.21.15.161:0/2679925990 conn(0x7f7a8806b260 msgr2=0x7f7a8806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.758+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/2679925990 shutdown_connections 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.758+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/2679925990 wait complete. 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.759+0000 7f7a8ed2e640 1 Processor -- start 2024-03-23T14:36:15.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.759+0000 7f7a8ed2e640 1 -- start start 2024-03-23T14:36:15.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8ed2e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a880d03e0 0x7f7a881513b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8ed2e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a881454f0 0x7f7a88140ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8ed2e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a88160390 0x7f7a880b3570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8ed2e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881f29c0 con 0x7f7a880d03e0 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8ed2e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7a881f2e50 con 0x7f7a881454f0 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8ed2e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7a881f1db0 con 0x7f7a88160390 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a880d03e0 0x7f7a881513b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a880d03e0 0x7f7a881513b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36540/0 (socket says 172.21.15.161:36540) 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 -- 172.21.15.161:0/92574240 learned_addr learned my addr 172.21.15.161:0/92574240 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a87fff640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a881454f0 0x7f7a88140ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8d2cb640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a88160390 0x7f7a880b3570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 -- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a88160390 msgr2=0x7f7a880b3570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a88160390 0x7f7a880b3570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 -- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a881454f0 msgr2=0x7f7a88140ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a881454f0 0x7f7a88140ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 -- 172.21.15.161:0/92574240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a80009a70 con 0x7f7a880d03e0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a87fff640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a881454f0 0x7f7a88140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8caca640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a880d03e0 0x7f7a881513b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c400 tx=0x7f7a7c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a857fa640 1 -- 172.21.15.161:0/92574240 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c0052d0 con 0x7f7a880d03e0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a857fa640 1 -- 172.21.15.161:0/92574240 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a7c005450 con 0x7f7a880d03e0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a881f2090 con 0x7f7a880d03e0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.760+0000 7f7a857fa640 1 -- 172.21.15.161:0/92574240 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a7c007260 con 0x7f7a880d03e0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.761+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a881f2650 con 0x7f7a880d03e0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.762+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a8804f4e0 con 0x7f7a880d03e0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.762+0000 7f7a857fa640 1 -- 172.21.15.161:0/92574240 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7a7c0073c0 con 0x7f7a880d03e0 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.762+0000 7f7a857fa640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a6c06b810 0x7f7a6c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:15.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.762+0000 7f7a857fa640 1 -- 172.21.15.161:0/92574240 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f7a7c089550 con 0x7f7a880d03e0 2024-03-23T14:36:15.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.762+0000 7f7a87fff640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a6c06b810 0x7f7a6c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:15.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.763+0000 7f7a87fff640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a6c06b810 0x7f7a6c06dcd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7a8806bd00 tx=0x7f7a74007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:15.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:15.767+0000 7f7a857fa640 1 -- 172.21.15.161:0/92574240 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7a7c054020 con 0x7f7a880d03e0 2024-03-23T14:36:15.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.002+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f7a8806ba50 con 0x7f7a880d03e0 2024-03-23T14:36:15.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.004+0000 7f7a857fa640 1 -- 172.21.15.161:0/92574240 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v11) ==== 190+0+147 (secure 0 0 0) 0x7f7a7c0576e0 con 0x7f7a880d03e0 2024-03-23T14:36:15.999 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:36:16.000 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:36:16.000 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-23T14:36:16.000 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-23T14:36:16.000 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-23T14:36:16.000 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a6c06b810 msgr2=0x7f7a6c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a6c06b810 0x7f7a6c06dcd0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f7a8806bd00 tx=0x7f7a74007d30 comp rx=0 tx=0).stop 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a880d03e0 msgr2=0x7f7a881513b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a880d03e0 0x7f7a881513b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f7a7c00c400 tx=0x7f7a7c00c8c0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 shutdown_connections 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7a6c06b810 0x7f7a6c06dcd0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7a88160390 0x7f7a880b3570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7a881454f0 0x7f7a88140ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 --2- 172.21.15.161:0/92574240 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7a880d03e0 0x7f7a881513b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 >> 172.21.15.161:0/92574240 conn(0x7f7a8806b260 msgr2=0x7f7a8815c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 shutdown_connections 2024-03-23T14:36:16.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.006+0000 7f7a8ed2e640 1 -- 172.21.15.161:0/92574240 wait complete. 2024-03-23T14:36:16.013 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:36:16.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.117+0000 7f532a842640 1 Processor -- start 2024-03-23T14:36:16.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.117+0000 7f532a842640 1 -- start start 2024-03-23T14:36:16.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f532a842640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f532415bd50 0x7f532415c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f532a842640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f53240d18a0 0x7f53240d1d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f532a842640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53240d2290 con 0x7f53240cf8a0 2024-03-23T14:36:16.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f532a842640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f53240d0b90 con 0x7f53240d18a0 2024-03-23T14:36:16.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f532a842640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f53240d05d0 con 0x7f532415bd50 2024-03-23T14:36:16.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f53237fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f53240d18a0 0x7f53240d1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f53237fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f53240d18a0 0x7f53240d1d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36552/0 (socket says 172.21.15.161:36552) 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f5328ddf640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f53240cf8a0 0x7f53240cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45486/0 (socket says 172.21.15.161:45486) 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f53237fe640 1 -- 172.21.15.161:0/1737161371 learned_addr learned my addr 172.21.15.161:0/1737161371 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.118+0000 7f5323fff640 1 --2- 172.21.15.161:0/1737161371 >> v2:172.21.15.183:3300/0 conn(0x7f532415bd50 0x7f532415c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.119+0000 7f53237fe640 1 -- 172.21.15.161:0/1737161371 >> v2:172.21.15.183:3300/0 conn(0x7f532415bd50 msgr2=0x7f532415c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.119+0000 7f53237fe640 1 --2- 172.21.15.161:0/1737161371 >> v2:172.21.15.183:3300/0 conn(0x7f532415bd50 0x7f532415c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.119+0000 7f53237fe640 1 -- 172.21.15.161:0/1737161371 >> v1:172.21.15.161:6789/0 conn(0x7f53240cf8a0 legacy=0x7f53240cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.119+0000 7f53237fe640 1 -- 172.21.15.161:0/1737161371 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53240d08b0 con 0x7f53240d18a0 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.119+0000 7f5322ffd640 1 -- 172.21.15.161:0/1737161371 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1333193008 0 0) 0x7f53240d2290 con 0x7f53240cf8a0 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.119+0000 7f53237fe640 1 --2- 172.21.15.161:0/1737161371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f53240d1d50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f53240cf780 tx=0x7f531802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=902ed61b251c143e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.119+0000 7f5322ffd640 1 -- 172.21.15.161:0/1737161371 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53180317e0 con 0x7f53240d18a0 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.119+0000 7f5322ffd640 1 -- 172.21.15.161:0/1737161371 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5318031940 con 0x7f53240d18a0 2024-03-23T14:36:16.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f5322ffd640 1 -- 172.21.15.161:0/1737161371 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318031cd0 con 0x7f53240d18a0 2024-03-23T14:36:16.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f532a842640 1 -- 172.21.15.161:0/1737161371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 msgr2=0x7f53240d1d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f532a842640 1 --2- 172.21.15.161:0/1737161371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f53240d1d50 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f53240cf780 tx=0x7f531802b880 comp rx=0 tx=0).stop 2024-03-23T14:36:16.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f532a842640 1 -- 172.21.15.161:0/1737161371 shutdown_connections 2024-03-23T14:36:16.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f532a842640 1 --2- 172.21.15.161:0/1737161371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f53240d1d50 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f532a842640 1 --2- 172.21.15.161:0/1737161371 >> v2:172.21.15.183:3300/0 conn(0x7f532415bd50 0x7f532415c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f532a842640 1 -- 172.21.15.161:0/1737161371 >> 172.21.15.161:0/1737161371 conn(0x7f532409aa40 msgr2=0x7f532409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f532a842640 1 -- 172.21.15.161:0/1737161371 shutdown_connections 2024-03-23T14:36:16.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.120+0000 7f532a842640 1 -- 172.21.15.161:0/1737161371 wait complete. 2024-03-23T14:36:16.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.121+0000 7f532a842640 1 Processor -- start 2024-03-23T14:36:16.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.121+0000 7f532a842640 1 -- start start 2024-03-23T14:36:16.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f532a842640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53240cf8a0 0x7f5324159d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f532a842640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f532415cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f532a842640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f532415bd50 0x7f53240c7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f532a842640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f53241f4920 con 0x7f53240d18a0 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f532a842640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f53241f3ff0 con 0x7f53240cf8a0 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f532a842640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f53241f42d0 con 0x7f532415bd50 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f53237fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f532415cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f53237fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f532415cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36558/0 (socket says 172.21.15.161:36558) 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f53237fe640 1 -- 172.21.15.161:0/1076473220 learned_addr learned my addr 172.21.15.161:0/1076473220 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f5323fff640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53240cf8a0 0x7f5324159d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.122+0000 7f5328ddf640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f532415bd50 0x7f53240c7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f53237fe640 1 -- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f532415bd50 msgr2=0x7f53240c7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f53237fe640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f532415bd50 0x7f53240c7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f53237fe640 1 -- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53240cf8a0 msgr2=0x7f5324159d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f53237fe640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53240cf8a0 0x7f5324159d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f53237fe640 1 -- 172.21.15.161:0/1076473220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53180094d0 con 0x7f53240d18a0 2024-03-23T14:36:16.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f53237fe640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f532415cb40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5318000c00 tx=0x7f53180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f5320ff9640 1 -- 172.21.15.161:0/1076473220 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318032880 con 0x7f53240d18a0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f5320ff9640 1 -- 172.21.15.161:0/1076473220 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f53180329e0 con 0x7f53240d18a0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f5320ff9640 1 -- 172.21.15.161:0/1076473220 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5318032d90 con 0x7f53240d18a0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53241f45b0 con 0x7f53240d18a0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.123+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53241f3ca0 con 0x7f53240d18a0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.124+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f532409fcf0 con 0x7f53240d18a0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.124+0000 7f5320ff9640 1 -- 172.21.15.161:0/1076473220 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f531804e450 con 0x7f53240d18a0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.125+0000 7f5320ff9640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f531006b810 0x7f531006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.125+0000 7f5320ff9640 1 -- 172.21.15.161:0/1076473220 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f53180ac1d0 con 0x7f53240d18a0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.125+0000 7f5323fff640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f531006b810 0x7f531006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.125+0000 7f5323fff640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f531006b810 0x7f531006dcd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5314009aa0 tx=0x7f5314009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.129+0000 7f5320ff9640 1 -- 172.21.15.161:0/1076473220 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5318076dc0 con 0x7f53240d18a0 2024-03-23T14:36:16.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.337+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f532404f550 con 0x7f53240d18a0 2024-03-23T14:36:16.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.337+0000 7f5320ff9640 1 -- 172.21.15.161:0/1076473220 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+5180 (secure 0 0 0) 0x7f531807a480 con 0x7f53240d18a0 2024-03-23T14:36:16.333 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/626265302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":626265302},{"type":"v1","addr":"172.21.15.183:6837","nonce":626265302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5220,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/208344896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":208344896},{"type":"v1","addr":"172.21.15.161:6833","nonce":208344896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/82303731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":82303731},{"type":"v1","addr":"172.21.15.161:6835","nonce":82303731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:06.267736+0000","modified":"2024-03-23T14:36:09.297699+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5235":{"gid":5235,"name":"d","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6836/2372368263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2372368263},{"type":"v1","addr":"172.21.15.183:6836","nonce":2372368263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5235,"qdb_cluster":[5235]},"id":2}]} 2024-03-23T14:36:16.333 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 15 2024-03-23T14:36:16.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f531006b810 msgr2=0x7f531006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f531006b810 0x7f531006dcd0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f5314009aa0 tx=0x7f5314009190 comp rx=0 tx=0).stop 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 msgr2=0x7f532415cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f532415cb40 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f5318000c00 tx=0x7f53180046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 shutdown_connections 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f531006b810 0x7f531006dcd0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f532415bd50 0x7f53240c7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f53240d18a0 0x7f532415cb40 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 --2- 172.21.15.161:0/1076473220 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f53240cf8a0 0x7f5324159d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 >> 172.21.15.161:0/1076473220 conn(0x7f532409aa40 msgr2=0x7f53240c6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 shutdown_connections 2024-03-23T14:36:16.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.339+0000 7f532a842640 1 -- 172.21.15.161:0/1076473220 wait complete. 2024-03-23T14:36:16.748 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T14:36:06.267736+0000', 'modified': '2024-03-23T14:36:09.297699+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5235}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5235': {'gid': 5235, 'name': 'd', 'rank': 0, 'incarnation': 11, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.183:6836/2372368263', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.183:6834', 'nonce': 2372368263}, {'type': 'v1', 'addr': '172.21.15.183:6836', 'nonce': 2372368263}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [5], 'metadata_pool': 4, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5235, 'qdb_cluster': [5235]} 2024-03-23T14:36:16.749 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T14:36:16.749 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-03-23T14:36:16.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.857+0000 7fbbf5bc9640 1 Processor -- start 2024-03-23T14:36:16.853 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.857+0000 7fbbf5bc9640 1 -- start start 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbf5bc9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbbf0072100 0x7fbbf015b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbf5bc9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf015be70 con 0x7fbbf00724e0 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbf5bc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbbf015c150 con 0x7fbbf006eca0 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbf5bc9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbbf015c430 con 0x7fbbf0072100 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbef7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbbf0072100 0x7fbbf015b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbeeffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbbf00724e0 0x7fbbf006e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45490/0 (socket says 172.21.15.161:45490) 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbeeffd640 1 -- 172.21.15.161:0/1887097322 learned_addr learned my addr 172.21.15.161:0/1887097322 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbef7fe640 1 -- 172.21.15.161:0/1887097322 >> v1:172.21.15.183:6789/0 conn(0x7fbbf006eca0 legacy=0x7fbbf00b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbef7fe640 1 -- 172.21.15.161:0/1887097322 >> v1:172.21.15.161:6789/0 conn(0x7fbbf00724e0 legacy=0x7fbbf006e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.854 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.858+0000 7fbbef7fe640 1 -- 172.21.15.161:0/1887097322 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbf00b2c70 con 0x7fbbf0072100 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.859+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/1887097322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf015b3f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00097a0 tx=0x7fbbe002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95e28577bdc961fa server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.859+0000 7fbbee7fc640 1 -- 172.21.15.161:0/1887097322 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0031830 con 0x7fbbf0072100 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.859+0000 7fbbee7fc640 1 -- 172.21.15.161:0/1887097322 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0031990 con 0x7fbbf0072100 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.859+0000 7fbbee7fc640 1 -- 172.21.15.161:0/1887097322 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe0031d20 con 0x7fbbf0072100 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.859+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/1887097322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 msgr2=0x7fbbf015b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.859+0000 7fbbf5bc9640 1 --2- 172.21.15.161:0/1887097322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf015b3f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fbbe00097a0 tx=0x7fbbe002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.859+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/1887097322 shutdown_connections 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.859+0000 7fbbf5bc9640 1 --2- 172.21.15.161:0/1887097322 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf015b3f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.860+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/1887097322 >> 172.21.15.161:0/1887097322 conn(0x7fbbf00b17f0 msgr2=0x7fbbf00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.860+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/1887097322 shutdown_connections 2024-03-23T14:36:16.856 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.860+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/1887097322 wait complete. 2024-03-23T14:36:16.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.861+0000 7fbbf5bc9640 1 Processor -- start 2024-03-23T14:36:16.857 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.861+0000 7fbbf5bc9640 1 -- start start 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbf5bc9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbbf006eca0 0x7fbbf0153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbf5bc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf0142650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbf5bc9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbf00724e0 0x7fbbf015ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbf5bc9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0142b90 con 0x7fbbf0072100 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbf5bc9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbbf0153a80 con 0x7fbbf00724e0 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbf5bc9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fbbf0153d60 con 0x7fbbf006eca0 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf0142650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf0142650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36572/0 (socket says 172.21.15.161:36572) 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 -- 172.21.15.161:0/3632545529 learned_addr learned my addr 172.21.15.161:0/3632545529 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:16.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbbf006eca0 0x7fbbf0153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeffff640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbf00724e0 0x7fbbf015ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 -- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbbf006eca0 msgr2=0x7fbbf0153410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbbf006eca0 0x7fbbf0153410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 -- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbf00724e0 msgr2=0x7fbbf015ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbf00724e0 0x7fbbf015ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 -- 172.21.15.161:0/3632545529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbe0009400 con 0x7fbbf0072100 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf0142650 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c390 tx=0x7fbbe400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbd7fff640 1 -- 172.21.15.161:0/3632545529 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4005ce0 con 0x7fbbf0072100 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbd7fff640 1 -- 172.21.15.161:0/3632545529 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe4005e40 con 0x7fbbf0072100 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbd7fff640 1 -- 172.21.15.161:0/3632545529 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe4007310 con 0x7fbbf0072100 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/3632545529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbf0154040 con 0x7fbbf0072100 2024-03-23T14:36:16.860 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.862+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/3632545529 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf0154600 con 0x7fbbf0072100 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.864+0000 7fbbd7fff640 1 -- 172.21.15.161:0/3632545529 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbbe4007470 con 0x7fbbf0072100 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.865+0000 7fbbd7fff640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbbdc06b840 0x7fbbdc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.865+0000 7fbbd7fff640 1 -- 172.21.15.161:0/3632545529 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fbbe4089580 con 0x7fbbf0072100 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.865+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbbdc06b840 0x7fbbdc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.865+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbbdc06b840 0x7fbbdc06dd00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0036000 tx=0x7fbbe003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.865+0000 7fbbf87ad740 1 Processor -- start 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.865+0000 7fbbf87ad740 1 -- start start 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b44075610 0x555b440485b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 0x555b440783a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44048af0 0x555b440788e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x555b43fbc180 con 0x555b44075610 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555b43fb96b0 con 0x555b44048af0 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x555b43fb4e90 con 0x555b440759f0 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbef7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b44075610 0x555b440485b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbef7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b44075610 0x555b440485b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36582/0 (socket says 172.21.15.161:36582) 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbef7fe640 1 -- 172.21.15.161:0/2565527033 learned_addr learned my addr 172.21.15.161:0/2565527033 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 0x555b440783a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.866+0000 7fbbeffff640 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44048af0 0x555b440788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbef7fe640 1 -- 172.21.15.161:0/2565527033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 msgr2=0x555b440783a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 0x555b440783a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbef7fe640 1 -- 172.21.15.161:0/2565527033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44048af0 msgr2=0x555b440788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44048af0 0x555b440788e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbef7fe640 1 -- 172.21.15.161:0/2565527033 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b43f71d80 con 0x555b44075610 2024-03-23T14:36:16.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b44075610 0x555b440485b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009890 tx=0x7fbbe0030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5358c20902f40db1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbd57fa640 1 -- 172.21.15.161:0/2565527033 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe004a070 con 0x555b44075610 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbd57fa640 1 -- 172.21.15.161:0/2565527033 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0004030 con 0x555b44075610 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.867+0000 7fbbd57fa640 1 -- 172.21.15.161:0/2565527033 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe003f040 con 0x555b44075610 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2565527033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b44075610 msgr2=0x555b440485b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b44075610 0x555b440485b0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0009890 tx=0x7fbbe0030000 comp rx=0 tx=0).stop 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2565527033 shutdown_connections 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44048af0 0x555b440788e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 0x555b440783a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/2565527033 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b44075610 0x555b440485b0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2565527033 >> 172.21.15.161:0/2565527033 conn(0x555b4404a820 msgr2=0x555b440933a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2565527033 shutdown_connections 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2565527033 wait complete. 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 -- start start 2024-03-23T14:36:16.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.868+0000 7fbbf87ad740 1 Processor -- start 2024-03-23T14:36:16.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 0x555b44065150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b440759f0 0x555b44064880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b4407e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x555b43f59d50 con 0x555b4407b560 2024-03-23T14:36:16.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555b43c88310 con 0x555b440759f0 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x555b43f3a8f0 con 0x555b44075610 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbf87ad740 10 client.0 _ll_register_callbacks cb 0x555b43cc5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b4407e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b4407e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36592/0 (socket says 172.21.15.161:36592) 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbef7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 0x555b44065150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeffff640 1 -- 172.21.15.161:0/3943640509 learned_addr learned my addr 172.21.15.161:0/3943640509 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b440759f0 0x555b44064880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeffff640 1 -- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 msgr2=0x555b44065150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeffff640 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 0x555b44065150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeffff640 1 -- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b440759f0 msgr2=0x555b44064880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeffff640 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b440759f0 0x555b44064880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.869+0000 7fbbeffff640 1 -- 172.21.15.161:0/3943640509 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf00b2c70 con 0x555b4407b560 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbeffff640 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b4407e780 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80126c0 tx=0x7fbbe8012ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6d7a0bb9198279f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-23T14:36:16.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbce7fc640 1 -- 172.21.15.161:0/3943640509 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8003b80 con 0x555b4407b560 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbce7fc640 1 -- 172.21.15.161:0/3943640509 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fbbe8003ce0 con 0x555b4407b560 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 1 -- 172.21.15.161:0/3943640509 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x555b43cb46c0 con 0x555b4407b560 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbce7fc640 1 -- 172.21.15.161:0/3943640509 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fbbe8003ea0 con 0x555b4407b560 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 10 client.5442 fetch_fsmap learned FSMap version 15 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 1 -- 172.21.15.161:0/3943640509 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x555b43fb3510 con 0x555b4407b560 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbce7fc640 1 -- 172.21.15.161:0/3943640509 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x555b43cb46c0 con 0x555b4407b560 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbce7fc640 1 -- 172.21.15.161:0/3943640509 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fbbe80050d0 con 0x555b4407b560 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 10 client.5442 fetch_fsmap finished waiting for FSMap version 15 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 10 client.5442 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 20 client.5442 populate_metadata read hostname 'smithi161' 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] conn(0x555b44079540 0x555b44079920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 4 client.5442 mds_command: new command op to 5235 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbf87ad740 1 -- 172.21.15.161:0/3943640509 --> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555b43d05270 con 0x555b44079540 2024-03-23T14:36:16.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.870+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] conn(0x555b44079540 0x555b44079920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.871+0000 7fbbce7fc640 10 client.5442 ms_handle_connect on v2:172.21.15.183:6834/2372368263 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.871+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] conn(0x555b44079540 0x555b44079920 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=516d7a8a8f5d714 server_cookie=39e0c5b5c0260747 in_seq=0 out_seq=0 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbce7fc640 1 -- 172.21.15.161:0/3943640509 <== mds.0 v2:172.21.15.183:6834/2372368263 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x555b43d05270 con 0x555b44079540 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbce7fc640 10 client.5442 handle_command_reply: tid=0 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 client.5442 shutdown 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 -- 172.21.15.161:0/3943640509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 msgr2=0x555b4407e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b4407e780 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80126c0 tx=0x7fbbe8012ac0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 -- 172.21.15.161:0/3943640509 shutdown_connections 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] conn(0x555b44079540 0x555b44079920 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b4407e780 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b440759f0 0x555b44064880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbce7fc640 0 client.5442 ms_handle_reset on v2:172.21.15.183:6834/2372368263 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/3943640509 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 0x555b44065150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 -- 172.21.15.161:0/3943640509 >> 172.21.15.161:0/3943640509 conn(0x555b44048af0 msgr2=0x555b4404c5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 -- 172.21.15.161:0/3943640509 shutdown_connections 2024-03-23T14:36:16.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 1 -- 172.21.15.161:0/3943640509 wait complete. 2024-03-23T14:36:16.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.872+0000 7fbbf87ad740 20 client.5442 trim_cache size 0 max 16384 2024-03-23T14:36:16.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.890+0000 7fbbf87ad740 1 Processor -- start 2024-03-23T14:36:16.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.890+0000 7fbbf87ad740 1 -- start start 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 0x555b44095e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b440759f0 0x555b4407f150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b4407b560 0x555b4407f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x555b44049b30 con 0x555b440759f0 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555b4407eb40 con 0x555b4407b560 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x555b4407b230 con 0x555b44075610 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b440759f0 0x555b4407f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b440759f0 0x555b4407f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36594/0 (socket says 172.21.15.161:36594) 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 -- 172.21.15.161:0/2866127628 learned_addr learned my addr 172.21.15.161:0/2866127628 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 0x555b44095e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeffff640 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b4407b560 0x555b4407f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 -- 172.21.15.161:0/2866127628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 msgr2=0x555b44095e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 0x555b44095e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 -- 172.21.15.161:0/2866127628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b4407b560 msgr2=0x555b4407f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b4407b560 0x555b4407f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 -- 172.21.15.161:0/2866127628 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555b43f4fa80 con 0x555b440759f0 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.891+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b440759f0 0x555b4407f150 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4057dd0 tx=0x7fbbe40540e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99c3eb4db40d57d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbd4ff9640 1 -- 172.21.15.161:0/2866127628 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe40604a0 con 0x555b440759f0 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbd4ff9640 1 -- 172.21.15.161:0/2866127628 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe4055ee0 con 0x555b440759f0 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbd4ff9640 1 -- 172.21.15.161:0/2866127628 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe405e0b0 con 0x555b440759f0 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2866127628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b440759f0 msgr2=0x555b4407f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b440759f0 0x555b4407f150 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fbbe4057dd0 tx=0x7fbbe40540e0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2866127628 shutdown_connections 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b4407b560 0x555b4407f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b440759f0 0x555b4407f150 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/2866127628 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b44075610 0x555b44095e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2866127628 >> 172.21.15.161:0/2866127628 conn(0x555b44048af0 msgr2=0x555b4407d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2866127628 shutdown_connections 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.892+0000 7fbbf87ad740 1 -- 172.21.15.161:0/2866127628 wait complete. 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.893+0000 7fbbf87ad740 1 -- start start 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.893+0000 7fbbf87ad740 1 Processor -- start 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.893+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44075610 0x555b44079d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.893+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 0x555b440f2110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbf87ad740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b440f2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x555b440b6870 con 0x555b4407b560 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x555b44094fa0 con 0x555b44075610 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbf87ad740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x555b440b69e0 con 0x555b440759f0 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbf87ad740 10 client.0 _ll_register_callbacks cb 0x7fbbe80122c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b440f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b440f2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36596/0 (socket says 172.21.15.161:36596) 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbef7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44075610 0x555b44079d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 -- 172.21.15.161:0/1111943596 learned_addr learned my addr 172.21.15.161:0/1111943596 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeeffd640 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 0x555b440f2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 -- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 msgr2=0x555b440f2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 0x555b440f2110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 -- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44075610 msgr2=0x555b44079d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44075610 0x555b44079d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 -- 172.21.15.161:0/1111943596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbf0154600 con 0x555b4407b560 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbeffff640 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b440f2650 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80126c0 tx=0x7fbbe8015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71975660c6f457cd server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbce7fc640 1 -- 172.21.15.161:0/1111943596 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbe8003d80 con 0x555b4407b560 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbce7fc640 1 -- 172.21.15.161:0/1111943596 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fbbe8006b50 con 0x555b4407b560 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbf87ad740 1 -- 172.21.15.161:0/1111943596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbbe0004030 con 0x555b4407b560 2024-03-23T14:36:16.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbce7fc640 1 -- 172.21.15.161:0/1111943596 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=15) ==== 24+0+0 (secure 0 0 0) 0x7fbbe8006d10 con 0x555b4407b560 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbf87ad740 10 client.5448 fetch_fsmap learned FSMap version 15 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbf87ad740 1 -- 172.21.15.161:0/1111943596 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=15,monmap=2+}) -- 0x555b43f757e0 con 0x555b4407b560 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.894+0000 7fbbce7fc640 1 -- 172.21.15.161:0/1111943596 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbbe0004030 con 0x555b4407b560 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbce7fc640 1 -- 172.21.15.161:0/1111943596 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 15) ==== 3494+0+0 (secure 0 0 0) 0x7fbbe80104c0 con 0x555b4407b560 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 10 client.5448 fetch_fsmap finished waiting for FSMap version 15 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 10 client.5448 resolve_mds: resolved name 'd' to daemon mds.d 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 20 client.5448 populate_metadata read hostname 'smithi161' 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] conn(0x555b44110620 0x555b44110a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 4 client.5448 mds_command: new command op to 5235 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 -- 172.21.15.161:0/1111943596 --> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] -- command(tid 0: {"prefix": "status"}) -- 0x555b43cf1610 con 0x555b44110620 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] conn(0x555b44110620 0x555b44110a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbce7fc640 10 client.5448 ms_handle_connect on v2:172.21.15.183:6834/2372368263 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbef7fe640 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] conn(0x555b44110620 0x555b44110a00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45727c5ca8bc9612 server_cookie=f5eb2de3ee7abe4b in_seq=0 out_seq=0 2024-03-23T14:36:16.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbce7fc640 1 -- 172.21.15.161:0/1111943596 <== mds.0 v2:172.21.15.183:6834/2372368263 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x555b43cf1610 con 0x555b44110620 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbce7fc640 10 client.5448 handle_command_reply: tid=0 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 client.5448 shutdown 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 -- 172.21.15.161:0/1111943596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 msgr2=0x555b440f2650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b440f2650 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fbbe80126c0 tx=0x7fbbe8015000 comp rx=0 tx=0).stop 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 -- 172.21.15.161:0/1111943596 shutdown_connections 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:6834/2372368263,v1:172.21.15.183:6836/2372368263] conn(0x555b44110620 0x555b44110a00 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x555b4407b560 0x555b440f2650 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x555b440759f0 0x555b440f2110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbce7fc640 0 client.5448 ms_handle_reset on v2:172.21.15.183:6834/2372368263 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.895+0000 7fbbf87ad740 1 --2- 172.21.15.161:0/1111943596 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x555b44075610 0x555b44079d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf87ad740 1 -- 172.21.15.161:0/1111943596 >> 172.21.15.161:0/1111943596 conn(0x555b44048af0 msgr2=0x555b44043810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf87ad740 1 -- 172.21.15.161:0/1111943596 shutdown_connections 2024-03-23T14:36:16.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf87ad740 1 -- 172.21.15.161:0/1111943596 wait complete. 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf87ad740 20 client.5448 trim_cache size 0 max 16384 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "9a16b48f-76af-4198-8fe3-d2c20a776a3e", 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5235, 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 9.9563600979999993, 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 15, 2024-03-23T14:36:16.899 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 33, 2024-03-23T14:36:16.900 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 33, 2024-03-23T14:36:16.900 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 16.855142665999999 2024-03-23T14:36:16.900 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbbdc06b840 msgr2=0x7fbbdc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbbdc06b840 0x7fbbdc06dd00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fbbe0036000 tx=0x7fbbe003c040 comp rx=0 tx=0).stop 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/3632545529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 msgr2=0x7fbbf0142650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf0142650 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fbbe400c390 tx=0x7fbbe400c850 comp rx=0 tx=0).stop 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/3632545529 shutdown_connections 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbbdc06b840 0x7fbbdc06dd00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbbf00724e0 0x7fbbf015ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbbf0072100 0x7fbbf0142650 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 --2- 172.21.15.161:0/3632545529 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbbf006eca0 0x7fbbf0153410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/3632545529 >> 172.21.15.161:0/3632545529 conn(0x7fbbf00b17f0 msgr2=0x7fbbf00d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/3632545529 shutdown_connections 2024-03-23T14:36:16.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:16.896+0000 7fbbf5bc9640 1 -- 172.21.15.161:0/3632545529 wait complete. 2024-03-23T14:36:16.903 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:36:17.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.008+0000 7f6d405bc640 1 Processor -- start 2024-03-23T14:36:17.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.008+0000 7f6d405bc640 1 -- start start 2024-03-23T14:36:17.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d405bc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6d3806eb30 0x7f6d380b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:17.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d405bc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6d380b4260 0x7f6d3815b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:17.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d405bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d380b3610 0x7f6d380b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:17.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d405bc640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f6d380b3fe0 con 0x7f6d3806eb30 2024-03-23T14:36:17.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d405bc640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f6d3815ab90 con 0x7f6d380b4260 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d405bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f6d380713a0 con 0x7f6d380b3610 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3e358640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6d3806eb30 0x7f6d380b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3e358640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f6d3806eb30 0x7f6d380b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36612/0 (socket says 172.21.15.161:36612) 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3e358640 1 -- 172.21.15.161:0/1244820797 learned_addr learned my addr 172.21.15.161:0/1244820797 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3db57640 1 --2- 172.21.15.161:0/1244820797 >> v2:172.21.15.183:3300/0 conn(0x7f6d380b4260 0x7f6d3815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3eb59640 1 --2- 172.21.15.161:0/1244820797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d380b3610 0x7f6d380b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3e358640 1 -- 172.21.15.161:0/1244820797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d380b3610 msgr2=0x7f6d380b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3e358640 1 --2- 172.21.15.161:0/1244820797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d380b3610 0x7f6d380b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3e358640 1 -- 172.21.15.161:0/1244820797 >> v2:172.21.15.183:3300/0 conn(0x7f6d380b4260 msgr2=0x7f6d3815b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3e358640 1 --2- 172.21.15.161:0/1244820797 >> v2:172.21.15.183:3300/0 conn(0x7f6d380b4260 0x7f6d3815b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.009+0000 7f6d3e358640 1 -- 172.21.15.161:0/1244820797 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d38070b90 con 0x7f6d3806eb30 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.010+0000 7f6d3e358640 1 --2- 172.21.15.161:0/1244820797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d3806eb30 0x7f6d380b4870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6d380b3560 tx=0x7f6d3002b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1251998e0d2a617d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.010+0000 7f6d3d356640 1 -- 172.21.15.161:0/1244820797 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30031640 con 0x7f6d3806eb30 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.010+0000 7f6d3d356640 1 -- 172.21.15.161:0/1244820797 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d300317a0 con 0x7f6d3806eb30 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.010+0000 7f6d3d356640 1 -- 172.21.15.161:0/1244820797 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d30031b30 con 0x7f6d3806eb30 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 -- 172.21.15.161:0/1244820797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d3806eb30 msgr2=0x7f6d380b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1244820797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d3806eb30 0x7f6d380b4870 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f6d380b3560 tx=0x7f6d3002b8f0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 -- 172.21.15.161:0/1244820797 shutdown_connections 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1244820797 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d380b3610 0x7f6d380b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1244820797 >> v2:172.21.15.183:3300/0 conn(0x7f6d380b4260 0x7f6d3815b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1244820797 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d3806eb30 0x7f6d380b4870 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 -- 172.21.15.161:0/1244820797 >> 172.21.15.161:0/1244820797 conn(0x7f6d380b17f0 msgr2=0x7f6d380b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:17.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 -- 172.21.15.161:0/1244820797 shutdown_connections 2024-03-23T14:36:17.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.011+0000 7f6d405bc640 1 -- 172.21.15.161:0/1244820797 wait complete. 2024-03-23T14:36:17.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.012+0000 7f6d405bc640 1 Processor -- start 2024-03-23T14:36:17.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.012+0000 7f6d405bc640 1 -- start start 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d405bc640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d3806eb30 0x7f6d38152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d405bc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d380b3610 0x7f6d3815e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d405bc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d380b4260 0x7f6d38159630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d405bc640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6d38154be0 con 0x7f6d380b4260 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d405bc640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f6d38154620 con 0x7f6d380b3610 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d405bc640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f6d38154900 con 0x7f6d3806eb30 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d380b4260 0x7f6d38159630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d380b4260 0x7f6d38159630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36618/0 (socket says 172.21.15.161:36618) 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 -- 172.21.15.161:0/1747597593 learned_addr learned my addr 172.21.15.161:0/1747597593 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:17.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3db57640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d380b3610 0x7f6d3815e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3e358640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d3806eb30 0x7f6d38152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 -- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d3806eb30 msgr2=0x7f6d38152b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d3806eb30 0x7f6d38152b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 -- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d380b3610 msgr2=0x7f6d3815e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d380b3610 0x7f6d3815e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 -- 172.21.15.161:0/1747597593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d300094d0 con 0x7f6d380b4260 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d3eb59640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d380b4260 0x7f6d38159630 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800e780 tx=0x7f6d2800ec40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.013+0000 7f6d36ffd640 1 -- 172.21.15.161:0/1747597593 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800c730 con 0x7f6d380b4260 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.014+0000 7f6d36ffd640 1 -- 172.21.15.161:0/1747597593 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6d2800c890 con 0x7f6d380b4260 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.014+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d38158e30 con 0x7f6d380b4260 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.014+0000 7f6d36ffd640 1 -- 172.21.15.161:0/1747597593 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6d2800cc40 con 0x7f6d380b4260 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.014+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d38153040 con 0x7f6d380b4260 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.015+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d38158c90 con 0x7f6d380b4260 2024-03-23T14:36:17.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.015+0000 7f6d36ffd640 1 -- 172.21.15.161:0/1747597593 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6d28013470 con 0x7f6d380b4260 2024-03-23T14:36:17.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.015+0000 7f6d36ffd640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6d2406b810 0x7f6d2406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:17.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.015+0000 7f6d36ffd640 1 -- 172.21.15.161:0/1747597593 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7f6d2808a420 con 0x7f6d380b4260 2024-03-23T14:36:17.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.015+0000 7f6d3e358640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6d2406b810 0x7f6d2406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:17.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.016+0000 7f6d3e358640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6d2406b810 0x7f6d2406dcd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6d30036000 tx=0x7f6d3003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:17.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.020+0000 7f6d36ffd640 1 -- 172.21.15.161:0/1747597593 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6d38158c90 con 0x7f6d380b4260 2024-03-23T14:36:17.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.230+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f6d38158c90 con 0x7f6d380b4260 2024-03-23T14:36:17.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.230+0000 7f6d36ffd640 1 -- 172.21.15.161:0/1747597593 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v15) ==== 53+0+73 (secure 0 0 0) 0x7f6d38158c90 con 0x7f6d380b4260 2024-03-23T14:36:17.226 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T14:36:17.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6d2406b810 msgr2=0x7f6d2406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:17.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6d2406b810 0x7f6d2406dcd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f6d30036000 tx=0x7f6d3003c040 comp rx=0 tx=0).stop 2024-03-23T14:36:17.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d380b4260 msgr2=0x7f6d38159630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d380b4260 0x7f6d38159630 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f6d2800e780 tx=0x7f6d2800ec40 comp rx=0 tx=0).stop 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 shutdown_connections 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f6d2406b810 0x7f6d2406dcd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6d380b4260 0x7f6d38159630 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6d380b3610 0x7f6d3815e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 --2- 172.21.15.161:0/1747597593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6d3806eb30 0x7f6d38152b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 >> 172.21.15.161:0/1747597593 conn(0x7f6d380b17f0 msgr2=0x7f6d3815cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 shutdown_connections 2024-03-23T14:36:17.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17.232+0000 7f6d405bc640 1 -- 172.21.15.161:0/1747597593 wait complete. 2024-03-23T14:36:17.239 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:36:17.239 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:36:17.239 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-23T14:36:17.239 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T14:36:17.239 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:17.239 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:36:17.240 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:36:17.240 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:36:17.240 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:17.240 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:17.240 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-23T14:36:17.255 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-23T14:36:17.256 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42693sec preferred_lft 42693sec 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c4a9:64ff:fe59:877a/64 scope link 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:17.257 INFO:teuthology.orchestra.run.smithi161.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:36:17.258 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:17.258 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c48:50ff:fe01:9f4e/64 scope link 2024-03-23T14:36:17.258 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:17.258 INFO:teuthology.orchestra.run.smithi161.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:36:17.258 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 62:02:d5:17:e7:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:17.258 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::6002:d5ff:fe17:e70d/64 scope link 2024-03-23T14:36:17.258 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:17.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:17.258 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:36:17.321 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-23T14:36:17.323 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:17.323 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-23T14:36:17.378 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-23T14:36:17.378 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:36:17.378 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:36:17.378 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:36:17.378 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:36:17.379 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:36:17.379 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:36:17.379 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:36:17.379 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:36:17.379 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:36:17.379 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:36:17.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:36:17.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:36:17.512 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:17.513 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:36:17.527 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T14:36:17.528 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T14:36:17.528 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:17.528 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:17.528 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-23T14:36:17.595 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T14:36:17.595 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:36:17.595 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:17.650 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:17.650 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:36:17.650 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:17.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:36:17.773 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:36:17.829 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:36:17.829 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:36:17.830 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:36:17.831 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:36:17.832 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:36:17.832 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:36:17.832 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:17.832 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:17.832 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:17.832 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:17.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:17.832 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:36:17.884 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T14:36:17.884 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T14:36:17.927 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:36:17.953 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:36:18.011 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:36:18.012 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:36:18.012 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:36:18.012 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:36:18.012 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:36:18.012 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:36:18.012 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:36:18.012 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:36:18.012 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:18.013 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:18.014 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:18.014 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:18.014 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:36:18.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:2024-03-23T14:36:18.017+0000 7f2f4a1ee480 -1 init, newargv = 0x5603af2c6d80 newargc=15 2024-03-23T14:36:18.015 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[51586]: starting ceph client 2024-03-23T14:36:18.017 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[51586]: starting fuse 2024-03-23T14:36:18.029 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:36:18.030 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-23T14:36:18.030 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:36:18.030 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-23T14:36:18.030 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-23T14:36:18.030 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:36:18.030 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-23T14:36:18.030 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:18.030 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-23T14:36:18.030 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-23T14:36:18.031 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:36:18.032 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:18.032 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.0")) 2024-03-23T14:36:18.032 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:36:18.150 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.0.51586.asok 2024-03-23T14:36:18.153 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:36:18.161 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.51586.asok 2024-03-23T14:36:18.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:18.161 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.51586.asok status 2024-03-23T14:36:18.296 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:36:18.296 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "0", 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "51586", 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5460, 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-23T14:36:18.297 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 5460 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3393783889 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3393783889 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:18.298 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.5460 192.168.0.1:0/3393783889", 2024-03-23T14:36:18.299 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/3393783889", 2024-03-23T14:36:18.299 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-23T14:36:18.299 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 15, 2024-03-23T14:36:18.299 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 33, 2024-03-23T14:36:18.299 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:36:18.299 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-23T14:36:18.299 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-23T14:36:18.299 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:36:18.305 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:18.305 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-23T14:36:18.305 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:18.320 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-23T14:36:18.320 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:18.320 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:18.320 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:18.387 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create data_new --pg_num_min 64 2024-03-23T14:36:18.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.536+0000 7fa1e2fc2640 1 Processor -- start 2024-03-23T14:36:18.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.536+0000 7fa1e2fc2640 1 -- start start 2024-03-23T14:36:18.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1e2fc2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa1dc071860 0x7fa1dc071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:18.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1e2fc2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1dc144ac0 con 0x7fa1dc1446e0 2024-03-23T14:36:18.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1e2fc2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1dc15b750 con 0x7fa1dc15bbe0 2024-03-23T14:36:18.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1e2fc2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa1dc072200 con 0x7fa1dc071860 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1e0d5e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa1dc1446e0 0x7fa1dc06de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45500/0 (socket says 172.21.15.161:45500) 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1e0d5e640 1 -- 172.21.15.161:0/3280399168 learned_addr learned my addr 172.21.15.161:0/3280399168 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1dbfff640 1 --2- 172.21.15.161:0/3280399168 >> v2:172.21.15.183:3300/0 conn(0x7fa1dc071860 0x7fa1dc071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1db7fe640 1 -- 172.21.15.161:0/3280399168 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2627380832 0 0) 0x7fa1dc144ac0 con 0x7fa1dc1446e0 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1db7fe640 1 -- 172.21.15.161:0/3280399168 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1c4003520 con 0x7fa1dc1446e0 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1dbfff640 1 -- 172.21.15.161:0/3280399168 >> v1:172.21.15.183:6789/0 conn(0x7fa1dc15bbe0 legacy=0x7fa1dc15c010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.537+0000 7fa1db7fe640 1 -- 172.21.15.161:0/3280399168 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3416988152 0 0) 0x7fa1dc15b750 con 0x7fa1dc15bbe0 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.538+0000 7fa1dbfff640 1 -- 172.21.15.161:0/3280399168 >> v1:172.21.15.161:6789/0 conn(0x7fa1dc1446e0 legacy=0x7fa1dc06de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:18.534 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.538+0000 7fa1dbfff640 1 -- 172.21.15.161:0/3280399168 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1dc06e550 con 0x7fa1dc071860 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.538+0000 7fa1dbfff640 1 --2- 172.21.15.161:0/3280399168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc071860 0x7fa1dc071cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00097a0 tx=0x7fa1d002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=89537661e7b6b6bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.538+0000 7fa1db7fe640 1 -- 172.21.15.161:0/3280399168 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0030610 con 0x7fa1dc071860 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.538+0000 7fa1db7fe640 1 -- 172.21.15.161:0/3280399168 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1d0030770 con 0x7fa1dc071860 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.538+0000 7fa1db7fe640 1 -- 172.21.15.161:0/3280399168 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0030ae0 con 0x7fa1dc071860 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.539+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3280399168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc071860 msgr2=0x7fa1dc071cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.539+0000 7fa1e2fc2640 1 --2- 172.21.15.161:0/3280399168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc071860 0x7fa1dc071cc0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00097a0 tx=0x7fa1d002b960 comp rx=0 tx=0).stop 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.539+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3280399168 shutdown_connections 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.539+0000 7fa1e2fc2640 1 --2- 172.21.15.161:0/3280399168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc071860 0x7fa1dc071cc0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.539+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3280399168 >> 172.21.15.161:0/3280399168 conn(0x7fa1dc06b080 msgr2=0x7fa1dc06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.539+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3280399168 shutdown_connections 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.539+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3280399168 wait complete. 2024-03-23T14:36:18.536 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.540+0000 7fa1e2fc2640 1 Processor -- start 2024-03-23T14:36:18.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.540+0000 7fa1e2fc2640 1 -- start start 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e2fc2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1dc071860 0x7fa1dc139270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e2fc2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1dc1446e0 0x7fa1dc15ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e2fc2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc15bbe0 0x7fa1dc15efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e2fc2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc15dae0 con 0x7fa1dc071860 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e2fc2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa1dc15ced0 con 0x7fa1dc15bbe0 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e2fc2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa1dc15d1b0 con 0x7fa1dc1446e0 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e0d5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1dc071860 0x7fa1dc139270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e0d5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1dc071860 0x7fa1dc139270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36628/0 (socket says 172.21.15.161:36628) 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e0d5e640 1 -- 172.21.15.161:0/3750014699 learned_addr learned my addr 172.21.15.161:0/3750014699 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1dbfff640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1dc1446e0 0x7fa1dc15ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e155f640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc15bbe0 0x7fa1dc15efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:18.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e0d5e640 1 -- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1dc1446e0 msgr2=0x7fa1dc15ea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e0d5e640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1dc1446e0 0x7fa1dc15ea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e0d5e640 1 -- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc15bbe0 msgr2=0x7fa1dc15efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e0d5e640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc15bbe0 0x7fa1dc15efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.541+0000 7fa1e0d5e640 1 -- 172.21.15.161:0/3750014699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d0009400 con 0x7fa1dc071860 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.542+0000 7fa1e0d5e640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1dc071860 0x7fa1dc139270 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00c430 tx=0x7fa1cc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.542+0000 7fa1d97fa640 1 -- 172.21.15.161:0/3750014699 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc005ce0 con 0x7fa1dc071860 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.542+0000 7fa1d97fa640 1 -- 172.21.15.161:0/3750014699 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa1cc005e40 con 0x7fa1dc071860 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.542+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1dc15d490 con 0x7fa1dc071860 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.542+0000 7fa1d97fa640 1 -- 172.21.15.161:0/3750014699 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc0072f0 con 0x7fa1dc071860 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.542+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1dc15df70 con 0x7fa1dc071860 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.543+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1dc0735e0 con 0x7fa1dc071860 2024-03-23T14:36:18.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.543+0000 7fa1d97fa640 1 -- 172.21.15.161:0/3750014699 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa1cc007450 con 0x7fa1dc071860 2024-03-23T14:36:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.544+0000 7fa1d97fa640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1c806b840 0x7fa1c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.544+0000 7fa1d97fa640 1 -- 172.21.15.161:0/3750014699 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(33..33 src has 1..33) ==== 5709+0+0 (secure 0 0 0) 0x7fa1cc0896e0 con 0x7fa1dc071860 2024-03-23T14:36:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.544+0000 7fa1dbfff640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1c806b840 0x7fa1c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:18.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.544+0000 7fa1dbfff640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1c806b840 0x7fa1c806dd00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0036000 tx=0x7fa1d0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:18.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.548+0000 7fa1d97fa640 1 -- 172.21.15.161:0/3750014699 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa1cc054240 con 0x7fa1dc071860 2024-03-23T14:36:18.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:18.720+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64} v 0) -- 0x7fa1dc04f550 con 0x7fa1dc071860 2024-03-23T14:36:19.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.010+0000 7fa1d97fa640 1 -- 172.21.15.161:0/3750014699 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "data_new", "pg_num_min": 64}]=0 pool 'data_new' created v34) ==== 124+0+0 (secure 0 0 0) 0x7fa1cc057900 con 0x7fa1dc071860 2024-03-23T14:36:19.247 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'data_new' created 2024-03-23T14:36:19.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.013+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1c806b840 msgr2=0x7fa1c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:19.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.013+0000 7fa1e2fc2640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1c806b840 0x7fa1c806dd00 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0036000 tx=0x7fa1d0008030 comp rx=0 tx=0).stop 2024-03-23T14:36:19.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.013+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1dc071860 msgr2=0x7fa1dc139270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:19.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.013+0000 7fa1e2fc2640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1dc071860 0x7fa1dc139270 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc00c430 tx=0x7fa1cc00c8f0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.014+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 shutdown_connections 2024-03-23T14:36:19.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.014+0000 7fa1e2fc2640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa1c806b840 0x7fa1c806dd00 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.014+0000 7fa1e2fc2640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa1dc15bbe0 0x7fa1dc15efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.014+0000 7fa1e2fc2640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa1dc1446e0 0x7fa1dc15ea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.014+0000 7fa1e2fc2640 1 --2- 172.21.15.161:0/3750014699 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa1dc071860 0x7fa1dc139270 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.014+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 >> 172.21.15.161:0/3750014699 conn(0x7fa1dc06b080 msgr2=0x7fa1dc0af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:19.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.014+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 shutdown_connections 2024-03-23T14:36:19.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.015+0000 7fa1e2fc2640 1 -- 172.21.15.161:0/3750014699 wait complete. 2024-03-23T14:36:19.250 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs data_new 2024-03-23T14:36:19.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.359+0000 7f5ab048f640 1 Processor -- start 2024-03-23T14:36:19.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.359+0000 7f5ab048f640 1 -- start start 2024-03-23T14:36:19.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5ab048f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa806e950 0x7f5aa815b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:19.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5ab048f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5aa815b5b0 0x7f5aa806d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:19.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5ab048f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aa815ba40 con 0x7f5aa81549c0 2024-03-23T14:36:19.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5ab048f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5aa80b2a10 con 0x7f5aa815b5b0 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5ab048f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa8071330 con 0x7f5aa806e950 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5aaea2c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5aa815b5b0 0x7f5aa806d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5aae22b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5aa81549c0 0x7f5aa8070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45506/0 (socket says 172.21.15.161:45506) 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5aae22b640 1 -- 172.21.15.161:0/2341102981 learned_addr learned my addr 172.21.15.161:0/2341102981 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.360+0000 7f5aada2a640 1 --2- 172.21.15.161:0/2341102981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa806e950 0x7f5aa815b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.361+0000 7f5aaea2c640 1 -- 172.21.15.161:0/2341102981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa806e950 msgr2=0x7f5aa815b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.361+0000 7f5aaea2c640 1 --2- 172.21.15.161:0/2341102981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa806e950 0x7f5aa815b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.361+0000 7f5aaea2c640 1 -- 172.21.15.161:0/2341102981 >> v1:172.21.15.161:6789/0 conn(0x7f5aa81549c0 legacy=0x7f5aa8070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.361+0000 7f5aaea2c640 1 -- 172.21.15.161:0/2341102981 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa806edd0 con 0x7f5aa815b5b0 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.361+0000 7f5aad229640 1 -- 172.21.15.161:0/2341102981 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144775426 0 0) 0x7f5aa4001490 con 0x7f5aa81549c0 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.361+0000 7f5aaea2c640 1 --2- 172.21.15.161:0/2341102981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa815b5b0 0x7f5aa806d6f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00097a0 tx=0x7f5aa002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2a8937ae8365d99 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:19.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.361+0000 7f5aad229640 1 -- 172.21.15.161:0/2341102981 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0031830 con 0x7f5aa815b5b0 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5aad229640 1 -- 172.21.15.161:0/2341102981 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5aa0031990 con 0x7f5aa815b5b0 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5aad229640 1 -- 172.21.15.161:0/2341102981 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5aa0031d20 con 0x7f5aa815b5b0 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5ab048f640 1 -- 172.21.15.161:0/2341102981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa815b5b0 msgr2=0x7f5aa806d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5ab048f640 1 --2- 172.21.15.161:0/2341102981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa815b5b0 0x7f5aa806d6f0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5aa00097a0 tx=0x7f5aa002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5ab048f640 1 -- 172.21.15.161:0/2341102981 shutdown_connections 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5ab048f640 1 --2- 172.21.15.161:0/2341102981 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa815b5b0 0x7f5aa806d6f0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5ab048f640 1 --2- 172.21.15.161:0/2341102981 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa806e950 0x7f5aa815b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5ab048f640 1 -- 172.21.15.161:0/2341102981 >> 172.21.15.161:0/2341102981 conn(0x7f5aa806b4a0 msgr2=0x7f5aa806b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5ab048f640 1 -- 172.21.15.161:0/2341102981 shutdown_connections 2024-03-23T14:36:19.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.362+0000 7f5ab048f640 1 -- 172.21.15.161:0/2341102981 wait complete. 2024-03-23T14:36:19.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.363+0000 7f5ab048f640 1 Processor -- start 2024-03-23T14:36:19.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.363+0000 7f5ab048f640 1 -- start start 2024-03-23T14:36:19.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5ab048f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aa806e950 0x7f5aa815c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:19.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5ab048f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa81549c0 0x7f5aa814c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5ab048f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa815b5b0 0x7f5aa814ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5ab048f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa815d590 con 0x7f5aa81549c0 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5ab048f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5aa814a620 con 0x7f5aa806e950 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5ab048f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5aa815cfd0 con 0x7f5aa815b5b0 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5aada2a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa81549c0 0x7f5aa814c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5aada2a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa81549c0 0x7f5aa814c8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36644/0 (socket says 172.21.15.161:36644) 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5aada2a640 1 -- 172.21.15.161:0/4163110104 learned_addr learned my addr 172.21.15.161:0/4163110104 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5aae22b640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aa806e950 0x7f5aa815c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.364+0000 7f5aaea2c640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa815b5b0 0x7f5aa814ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5aada2a640 1 -- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa815b5b0 msgr2=0x7f5aa814ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5aada2a640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa815b5b0 0x7f5aa814ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5aada2a640 1 -- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aa806e950 msgr2=0x7f5aa815c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5aada2a640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aa806e950 0x7f5aa815c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5aada2a640 1 -- 172.21.15.161:0/4163110104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aa0009400 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5aada2a640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa81549c0 0x7f5aa814c8e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e9b0 tx=0x7f5a9800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5a9effd640 1 -- 172.21.15.161:0/4163110104 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a98009670 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5a9effd640 1 -- 172.21.15.161:0/4163110104 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a9800c9a0 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5a9effd640 1 -- 172.21.15.161:0/4163110104 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a9800cd50 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aa815d2b0 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.365+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aa8152b20 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.366+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aa81526a0 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.366+0000 7f5a9effd640 1 -- 172.21.15.161:0/4163110104 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5a98013470 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.367+0000 7f5a9effd640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a8c06b810 0x7f5a8c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.367+0000 7f5a9effd640 1 -- 172.21.15.161:0/4163110104 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 6072+0+0 (secure 0 0 0) 0x7f5a9808a4d0 con 0x7f5aa81549c0 2024-03-23T14:36:19.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.367+0000 7f5aae22b640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a8c06b810 0x7f5a8c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.367+0000 7f5aae22b640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a8c06b810 0x7f5a8c06dcd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40012e0 tx=0x7f5aa4007f10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:19.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.371+0000 7f5a9effd640 1 -- 172.21.15.161:0/4163110104 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5aa81526a0 con 0x7f5aa81549c0 2024-03-23T14:36:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:19.580+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"} v 0) -- 0x7f5aa81526a0 con 0x7f5aa81549c0 2024-03-23T14:36:20.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.011+0000 7f5a9effd640 1 -- 172.21.15.161:0/4163110104 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "data_new"}]=0 added data pool 6 to fsmap v16) ==== 131+0+0 (secure 0 0 0) 0x7f5aa81526a0 con 0x7f5aa81549c0 2024-03-23T14:36:20.007 INFO:teuthology.orchestra.run.smithi161.stderr:added data pool 6 to fsmap 2024-03-23T14:36:20.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.016+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a8c06b810 msgr2=0x7f5a8c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.016+0000 7f5ab048f640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a8c06b810 0x7f5a8c06dcd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5aa40012e0 tx=0x7f5aa4007f10 comp rx=0 tx=0).stop 2024-03-23T14:36:20.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.016+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa81549c0 msgr2=0x7f5aa814c8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.016+0000 7f5ab048f640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa81549c0 0x7f5aa814c8e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5a9800e9b0 tx=0x7f5a9800ee70 comp rx=0 tx=0).stop 2024-03-23T14:36:20.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.019+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 shutdown_connections 2024-03-23T14:36:20.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.019+0000 7f5ab048f640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a8c06b810 0x7f5a8c06dcd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.019+0000 7f5ab048f640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aa815b5b0 0x7f5aa814ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.019+0000 7f5ab048f640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aa81549c0 0x7f5aa814c8e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.019+0000 7f5ab048f640 1 --2- 172.21.15.161:0/4163110104 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aa806e950 0x7f5aa815c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.019+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 >> 172.21.15.161:0/4163110104 conn(0x7f5aa806b4a0 msgr2=0x7f5aa814b970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:20.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.020+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 shutdown_connections 2024-03-23T14:36:20.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.020+0000 7f5ab048f640 1 -- 172.21.15.161:0/4163110104 wait complete. 2024-03-23T14:36:20.030 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:36:20.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.134+0000 7fb98e810640 1 Processor -- start 2024-03-23T14:36:20.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.134+0000 7fb98e810640 1 -- start start 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb98e810640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb98806eb30 0x7fb9880b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb98e810640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb9880b3610 0x7fb9880b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb98e810640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9880b3fa0 con 0x7fb9880b4260 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb98e810640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb98815ab90 con 0x7fb98806eb30 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb98e810640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fb9880713a0 con 0x7fb9880b3610 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb987fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb98806eb30 0x7fb9880b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb987fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb98806eb30 0x7fb9880b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36646/0 (socket says 172.21.15.161:36646) 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb987fff640 1 -- 172.21.15.161:0/1157012320 learned_addr learned my addr 172.21.15.161:0/1157012320 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb98cdad640 1 --2- 172.21.15.161:0/1157012320 >> v2:172.21.15.183:3300/0 conn(0x7fb9880b3610 0x7fb9880b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb987fff640 1 -- 172.21.15.161:0/1157012320 >> v2:172.21.15.183:3300/0 conn(0x7fb9880b3610 msgr2=0x7fb9880b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:20.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb987fff640 1 --2- 172.21.15.161:0/1157012320 >> v2:172.21.15.183:3300/0 conn(0x7fb9880b3610 0x7fb9880b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb987fff640 1 -- 172.21.15.161:0/1157012320 >> v1:172.21.15.183:6789/0 conn(0x7fb9880b4260 legacy=0x7fb98815b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.135+0000 7fb987fff640 1 -- 172.21.15.161:0/1157012320 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb988070b90 con 0x7fb98806eb30 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.136+0000 7fb987fff640 1 --2- 172.21.15.161:0/1157012320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb98806eb30 0x7fb9880b4870 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb98806c9b0 tx=0x7fb97802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=143d3e4a30947773 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.136+0000 7fb986ffd640 1 -- 172.21.15.161:0/1157012320 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978031870 con 0x7fb98806eb30 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.136+0000 7fb986ffd640 1 -- 172.21.15.161:0/1157012320 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb9780319d0 con 0x7fb98806eb30 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.136+0000 7fb986ffd640 1 -- 172.21.15.161:0/1157012320 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb978031d60 con 0x7fb98806eb30 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.137+0000 7fb98e810640 1 -- 172.21.15.161:0/1157012320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb98806eb30 msgr2=0x7fb9880b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.137+0000 7fb98e810640 1 --2- 172.21.15.161:0/1157012320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb98806eb30 0x7fb9880b4870 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fb98806c9b0 tx=0x7fb97802b910 comp rx=0 tx=0).stop 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.137+0000 7fb98e810640 1 -- 172.21.15.161:0/1157012320 shutdown_connections 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.137+0000 7fb98e810640 1 --2- 172.21.15.161:0/1157012320 >> v2:172.21.15.183:3300/0 conn(0x7fb9880b3610 0x7fb9880b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.137+0000 7fb98e810640 1 --2- 172.21.15.161:0/1157012320 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb98806eb30 0x7fb9880b4870 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.137+0000 7fb98e810640 1 -- 172.21.15.161:0/1157012320 >> 172.21.15.161:0/1157012320 conn(0x7fb9880b17f0 msgr2=0x7fb9880b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:20.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.137+0000 7fb98e810640 1 -- 172.21.15.161:0/1157012320 shutdown_connections 2024-03-23T14:36:20.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.137+0000 7fb98e810640 1 -- 172.21.15.161:0/1157012320 wait complete. 2024-03-23T14:36:20.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.138+0000 7fb98e810640 1 Processor -- start 2024-03-23T14:36:20.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.138+0000 7fb98e810640 1 -- start start 2024-03-23T14:36:20.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98e810640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98806eb30 0x7fb98814a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98e810640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9880b3610 0x7fb988149700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98e810640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9880b4260 0x7fb988168040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98e810640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb98814ab30 con 0x7fb9880b4260 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98e810640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb9880cd0a0 con 0x7fb98806eb30 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98e810640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb988149c40 con 0x7fb9880b3610 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98cdad640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9880b4260 0x7fb988168040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98cdad640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9880b4260 0x7fb988168040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:36660/0 (socket says 172.21.15.161:36660) 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98cdad640 1 -- 172.21.15.161:0/3963132689 learned_addr learned my addr 172.21.15.161:0/3963132689 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb9877fe640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9880b3610 0x7fb988149700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb987fff640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98806eb30 0x7fb98814a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98cdad640 1 -- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9880b3610 msgr2=0x7fb988149700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98cdad640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9880b3610 0x7fb988149700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98cdad640 1 -- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98806eb30 msgr2=0x7fb98814a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98cdad640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98806eb30 0x7fb98814a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.139+0000 7fb98cdad640 1 -- 172.21.15.161:0/3963132689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9780094d0 con 0x7fb9880b4260 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.140+0000 7fb98cdad640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9880b4260 0x7fb988168040 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb980009d20 tx=0x7fb98000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.140+0000 7fb984ff9640 1 -- 172.21.15.161:0/3963132689 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb980009b10 con 0x7fb9880b4260 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.140+0000 7fb984ff9640 1 -- 172.21.15.161:0/3963132689 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb98000f040 con 0x7fb9880b4260 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.140+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9880ce4e0 con 0x7fb9880b4260 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.140+0000 7fb984ff9640 1 -- 172.21.15.161:0/3963132689 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9800146c0 con 0x7fb9880b4260 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.140+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9880ccae0 con 0x7fb9880b4260 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.141+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb98804f550 con 0x7fb9880b4260 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.141+0000 7fb984ff9640 1 -- 172.21.15.161:0/3963132689 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb980014820 con 0x7fb9880b4260 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.141+0000 7fb984ff9640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb97406b840 0x7fb97406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.142+0000 7fb984ff9640 1 -- 172.21.15.161:0/3963132689 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6104+0+0 (secure 0 0 0) 0x7fb98008ace0 con 0x7fb9880b4260 2024-03-23T14:36:20.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.142+0000 7fb987fff640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb97406b840 0x7fb97406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.142+0000 7fb987fff640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb97406b840 0x7fb97406dd00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb978036000 tx=0x7fb97803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.146+0000 7fb984ff9640 1 -- 172.21.15.161:0/3963132689 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb9800556b0 con 0x7fb9880b4260 2024-03-23T14:36:20.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.353+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb9880b2ee0 con 0x7fb9880b4260 2024-03-23T14:36:20.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.354+0000 7fb984ff9640 1 -- 172.21.15.161:0/3963132689 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5182 (secure 0 0 0) 0x7fb980058d70 con 0x7fb9880b4260 2024-03-23T14:36:20.351 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:20.351 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/626265302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":626265302},{"type":"v1","addr":"172.21.15.183:6837","nonce":626265302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5220,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/208344896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":208344896},{"type":"v1","addr":"172.21.15.161:6833","nonce":208344896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/82303731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":82303731},{"type":"v1","addr":"172.21.15.161:6835","nonce":82303731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:06.267736+0000","modified":"2024-03-23T14:36:19.582504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5235":{"gid":5235,"name":"d","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6836/2372368263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2372368263},{"type":"v1","addr":"172.21.15.183:6836","nonce":2372368263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5235,"qdb_cluster":[5235]},"id":2}]} 2024-03-23T14:36:20.351 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 16 2024-03-23T14:36:20.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb97406b840 msgr2=0x7fb97406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb97406b840 0x7fb97406dd00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb978036000 tx=0x7fb97803c040 comp rx=0 tx=0).stop 2024-03-23T14:36:20.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9880b4260 msgr2=0x7fb988168040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9880b4260 0x7fb988168040 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fb980009d20 tx=0x7fb98000c490 comp rx=0 tx=0).stop 2024-03-23T14:36:20.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 shutdown_connections 2024-03-23T14:36:20.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb97406b840 0x7fb97406dd00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9880b4260 0x7fb988168040 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb9880b3610 0x7fb988149700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 --2- 172.21.15.161:0/3963132689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb98806eb30 0x7fb98814a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 >> 172.21.15.161:0/3963132689 conn(0x7fb9880b17f0 msgr2=0x7fb98815d0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:20.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 shutdown_connections 2024-03-23T14:36:20.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.356+0000 7fb98e810640 1 -- 172.21.15.161:0/3963132689 wait complete. 2024-03-23T14:36:20.363 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:20.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.468+0000 7f48d9332640 1 Processor -- start 2024-03-23T14:36:20.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.468+0000 7f48d9332640 1 -- start start 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.468+0000 7f48d9332640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f48d40d2690 0x7f48d40d1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.468+0000 7f48d9332640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d415ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.468+0000 7f48d9332640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48d40d06a0 con 0x7f48d4162970 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.468+0000 7f48d9332640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f48d40d0980 con 0x7f48d40d2690 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.468+0000 7f48d9332640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f48d40d0c60 con 0x7f48d40d2a70 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d2ffd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f48d40d2690 0x7f48d40d1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d37fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d415ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48cbfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f48d4162970 0x7f48d40cea60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:42916/0 (socket says 172.21.15.161:42916) 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48cbfff640 1 -- 172.21.15.161:0/597978507 learned_addr learned my addr 172.21.15.161:0/597978507 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d27fc640 1 -- 172.21.15.161:0/597978507 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141234729 0 0) 0x7f48d40d06a0 con 0x7f48d4162970 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d27fc640 1 -- 172.21.15.161:0/597978507 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f48c0003520 con 0x7f48d4162970 2024-03-23T14:36:20.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d37fe640 1 -- 172.21.15.161:0/597978507 >> v2:172.21.15.183:3300/0 conn(0x7f48d40d2690 msgr2=0x7f48d40d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d37fe640 1 --2- 172.21.15.161:0/597978507 >> v2:172.21.15.183:3300/0 conn(0x7f48d40d2690 0x7f48d40d1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d37fe640 1 -- 172.21.15.161:0/597978507 >> v1:172.21.15.183:6789/0 conn(0x7f48d4162970 legacy=0x7f48d40cea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d37fe640 1 -- 172.21.15.161:0/597978507 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48d40d0f40 con 0x7f48d40d2a70 2024-03-23T14:36:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.469+0000 7f48d2ffd640 1 --2- 172.21.15.161:0/597978507 >> v2:172.21.15.183:3300/0 conn(0x7f48d40d2690 0x7f48d40d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.470+0000 7f48d37fe640 1 --2- 172.21.15.161:0/597978507 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d415ee00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009d90 tx=0x7f48cc02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ba0ae78d5bc109f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.470+0000 7f48d27fc640 1 -- 172.21.15.161:0/597978507 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc02cc50 con 0x7f48d40d2a70 2024-03-23T14:36:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.470+0000 7f48d27fc640 1 -- 172.21.15.161:0/597978507 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48cc02cdb0 con 0x7f48d40d2a70 2024-03-23T14:36:20.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.470+0000 7f48d27fc640 1 -- 172.21.15.161:0/597978507 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc034640 con 0x7f48d40d2a70 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.470+0000 7f48d9332640 1 -- 172.21.15.161:0/597978507 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 msgr2=0x7f48d415ee00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.470+0000 7f48d9332640 1 --2- 172.21.15.161:0/597978507 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d415ee00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f48cc009d90 tx=0x7f48cc02bea0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.471+0000 7f48d9332640 1 -- 172.21.15.161:0/597978507 shutdown_connections 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.471+0000 7f48d9332640 1 --2- 172.21.15.161:0/597978507 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d415ee00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.471+0000 7f48d9332640 1 --2- 172.21.15.161:0/597978507 >> v2:172.21.15.183:3300/0 conn(0x7f48d40d2690 0x7f48d40d1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.471+0000 7f48d9332640 1 -- 172.21.15.161:0/597978507 >> 172.21.15.161:0/597978507 conn(0x7f48d409d1b0 msgr2=0x7f48d40d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.471+0000 7f48d9332640 1 -- 172.21.15.161:0/597978507 shutdown_connections 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.471+0000 7f48d9332640 1 -- 172.21.15.161:0/597978507 wait complete. 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.472+0000 7f48d9332640 1 Processor -- start 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.472+0000 7f48d9332640 1 -- start start 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.472+0000 7f48d9332640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48d40d2690 0x7f48d40b81e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.472+0000 7f48d9332640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d414d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.472+0000 7f48d9332640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d4162970 0x7f48d40b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.472+0000 7f48d9332640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f48d40b2040 con 0x7f48d4162970 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.472+0000 7f48d9332640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f48d40b3220 con 0x7f48d40d2690 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.472+0000 7f48d9332640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f48d40b2f80 con 0x7f48d40d2a70 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d2ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48d40d2690 0x7f48d40b81e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d4162970 0x7f48d40b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d2ffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48d40d2690 0x7f48d40b81e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:47744/0 (socket says 172.21.15.161:47744) 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d2ffd640 1 -- 172.21.15.161:0/1341255463 learned_addr learned my addr 172.21.15.161:0/1341255463 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48cbfff640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d414d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d37fe640 1 -- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 msgr2=0x7f48d414d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d37fe640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d414d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d37fe640 1 -- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48d40d2690 msgr2=0x7f48d40b81e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d37fe640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48d40d2690 0x7f48d40b81e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d37fe640 1 -- 172.21.15.161:0/1341255463 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48c4009400 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d2ffd640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48d40d2690 0x7f48d40b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d37fe640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d4162970 0x7f48d40b2a40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f48d40d0da0 tx=0x7f48cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48cb7fe640 1 -- 172.21.15.161:0/1341255463 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc009450 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48cb7fe640 1 -- 172.21.15.161:0/1341255463 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f48cc040450 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48cc009a70 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48cb7fe640 1 -- 172.21.15.161:0/1341255463 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48cc040820 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.473+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48d40b19f0 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.474+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48d40b21b0 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.475+0000 7f48cb7fe640 1 -- 172.21.15.161:0/1341255463 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f48cc040980 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.475+0000 7f48cb7fe640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f48b406b810 0x7f48b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.475+0000 7f48cb7fe640 1 -- 172.21.15.161:0/1341255463 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6104+0+0 (secure 0 0 0) 0x7f48cc0acb50 con 0x7f48d4162970 2024-03-23T14:36:20.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.475+0000 7f48d2ffd640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f48b406b810 0x7f48b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.474 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.476+0000 7f48d2ffd640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f48b406b810 0x7f48b406dcd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f48c4009fd0 tx=0x7f48c4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.479+0000 7f48cb7fe640 1 -- 172.21.15.161:0/1341255463 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f48cc077410 con 0x7f48d4162970 2024-03-23T14:36:20.647 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.652+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f48d40a1f70 con 0x7f48d4162970 2024-03-23T14:36:20.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.653+0000 7f48cb7fe640 1 -- 172.21.15.161:0/1341255463 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v35) ==== 74+0+18530 (secure 0 0 0) 0x7f48cc07aad0 con 0x7f48d4162970 2024-03-23T14:36:20.649 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:20.649 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":35,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:19.582302+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:20.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f48b406b810 msgr2=0x7f48b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f48b406b810 0x7f48b406dcd0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f48c4009fd0 tx=0x7f48c4009190 comp rx=0 tx=0).stop 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d4162970 msgr2=0x7f48d40b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d4162970 0x7f48d40b2a40 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f48d40d0da0 tx=0x7f48cc004a40 comp rx=0 tx=0).stop 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 shutdown_connections 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f48b406b810 0x7f48b406dcd0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f48d4162970 0x7f48d40b2a40 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f48d40d2a70 0x7f48d414d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 --2- 172.21.15.161:0/1341255463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f48d40d2690 0x7f48d40b81e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 >> 172.21.15.161:0/1341255463 conn(0x7f48d409d1b0 msgr2=0x7f48d40cea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 shutdown_connections 2024-03-23T14:36:20.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.655+0000 7f48d9332640 1 -- 172.21.15.161:0/1341255463 wait complete. 2024-03-23T14:36:20.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:20.662 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch layoutfile) 2024-03-23T14:36:20.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:20.682 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest/mnt.0 && exec mkdir layoutdir) 2024-03-23T14:36:20.740 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rw' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-23T14:36:20.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.891+0000 7fc0c37c7640 1 Processor -- start 2024-03-23T14:36:20.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.892+0000 7fc0c37c7640 1 -- start start 2024-03-23T14:36:20.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.892+0000 7fc0c37c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc0bc0d0380 0x7fc0bc165b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.892+0000 7fc0c37c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0bc15f480 con 0x7fc0bc1552f0 2024-03-23T14:36:20.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.892+0000 7fc0c37c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0bc1556d0 con 0x7fc0bc0b2aa0 2024-03-23T14:36:20.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.892+0000 7fc0c37c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc0bc06eee0 con 0x7fc0bc0d0380 2024-03-23T14:36:20.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0c1563640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc0bc1552f0 0x7fc0bc0b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49990/0 (socket says 172.21.15.161:49990) 2024-03-23T14:36:20.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0c1563640 1 -- 172.21.15.161:0/1635083474 learned_addr learned my addr 172.21.15.161:0/1635083474 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:20.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0c1d64640 1 --2- 172.21.15.161:0/1635083474 >> v2:172.21.15.183:3300/0 conn(0x7fc0bc0d0380 0x7fc0bc165b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0b3fff640 1 -- 172.21.15.161:0/1635083474 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1993169246 0 0) 0x7fc0bc15f480 con 0x7fc0bc1552f0 2024-03-23T14:36:20.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0b3fff640 1 -- 172.21.15.161:0/1635083474 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0a8003520 con 0x7fc0bc1552f0 2024-03-23T14:36:20.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0c1d64640 1 -- 172.21.15.161:0/1635083474 >> v1:172.21.15.183:6789/0 conn(0x7fc0bc0b2aa0 legacy=0x7fc0bc0cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0b3fff640 1 -- 172.21.15.161:0/1635083474 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 364598377 0 0) 0x7fc0bc1556d0 con 0x7fc0bc0b2aa0 2024-03-23T14:36:20.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0c1d64640 1 -- 172.21.15.161:0/1635083474 >> v1:172.21.15.161:6789/0 conn(0x7fc0bc1552f0 legacy=0x7fc0bc0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.893+0000 7fc0c1d64640 1 -- 172.21.15.161:0/1635083474 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0bc15e870 con 0x7fc0bc0d0380 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0c1d64640 1 --2- 172.21.15.161:0/1635083474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc165b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40097a0 tx=0x7fc0b402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aba3d788f4aa25da server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0b3fff640 1 -- 172.21.15.161:0/1635083474 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4030610 con 0x7fc0bc0d0380 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0b3fff640 1 -- 172.21.15.161:0/1635083474 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b4030770 con 0x7fc0bc0d0380 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0b3fff640 1 -- 172.21.15.161:0/1635083474 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4030ae0 con 0x7fc0bc0d0380 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0c37c7640 1 -- 172.21.15.161:0/1635083474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc0d0380 msgr2=0x7fc0bc165b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0c37c7640 1 --2- 172.21.15.161:0/1635083474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc165b60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40097a0 tx=0x7fc0b402b960 comp rx=0 tx=0).stop 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0c37c7640 1 -- 172.21.15.161:0/1635083474 shutdown_connections 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0c37c7640 1 --2- 172.21.15.161:0/1635083474 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc165b60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.894+0000 7fc0c37c7640 1 -- 172.21.15.161:0/1635083474 >> 172.21.15.161:0/1635083474 conn(0x7fc0bc06b260 msgr2=0x7fc0bc06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:20.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.895+0000 7fc0c37c7640 1 -- 172.21.15.161:0/1635083474 shutdown_connections 2024-03-23T14:36:20.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.895+0000 7fc0c37c7640 1 -- 172.21.15.161:0/1635083474 wait complete. 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c37c7640 1 Processor -- start 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c37c7640 1 -- start start 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c37c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0bc0b2aa0 0x7fc0bc14a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c37c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc148700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c37c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc1552f0 0x7fc0bc156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c37c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc157c20 con 0x7fc0bc0d0380 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c37c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc0bc148c40 con 0x7fc0bc1552f0 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c37c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc0bc156fa0 con 0x7fc0bc0b2aa0 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c0d62640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc148700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c0d62640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc148700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51474/0 (socket says 172.21.15.161:51474) 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.896+0000 7fc0c0d62640 1 -- 172.21.15.161:0/3551749635 learned_addr learned my addr 172.21.15.161:0/3551749635 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c1d64640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc1552f0 0x7fc0bc156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c1563640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0bc0b2aa0 0x7fc0bc14a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c0d62640 1 -- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0bc0b2aa0 msgr2=0x7fc0bc14a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c0d62640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0bc0b2aa0 0x7fc0bc14a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c0d62640 1 -- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc1552f0 msgr2=0x7fc0bc156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c0d62640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc1552f0 0x7fc0bc156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c0d62640 1 -- 172.21.15.161:0/3551749635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0b4009400 con 0x7fc0bc0d0380 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c0d62640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc148700 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac00e6e0 tx=0x7fc0ac00eba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0b1ffb640 1 -- 172.21.15.161:0/3551749635 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac005270 con 0x7fc0bc0d0380 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0b1ffb640 1 -- 172.21.15.161:0/3551749635 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0ac0053d0 con 0x7fc0bc0d0380 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0bc157280 con 0x7fc0bc0d0380 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0b1ffb640 1 -- 172.21.15.161:0/3551749635 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0ac00ca90 con 0x7fc0bc0d0380 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.897+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0bc157840 con 0x7fc0bc0d0380 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.898+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0bc15ee30 con 0x7fc0bc0d0380 2024-03-23T14:36:20.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.899+0000 7fc0b1ffb640 1 -- 172.21.15.161:0/3551749635 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc0ac00cbf0 con 0x7fc0bc0d0380 2024-03-23T14:36:20.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.899+0000 7fc0b1ffb640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0a006b840 0x7fc0a006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:20.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.899+0000 7fc0b1ffb640 1 -- 172.21.15.161:0/3551749635 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(35..35 src has 1..35) ==== 6104+0+0 (secure 0 0 0) 0x7fc0ac08c550 con 0x7fc0bc0d0380 2024-03-23T14:36:20.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.899+0000 7fc0c1563640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0a006b840 0x7fc0a006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:20.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.899+0000 7fc0c1563640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0a006b840 0x7fc0a006dd00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8005c00 tx=0x7fc0b8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:20.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:20.903+0000 7fc0b1ffb640 1 -- 172.21.15.161:0/3551749635 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc0bc15ee30 con 0x7fc0bc0d0380 2024-03-23T14:36:21.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.142+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7fc0bc157d60 con 0x7fc0bc0d0380 2024-03-23T14:36:21.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.146+0000 7fc0b1ffb640 1 -- 172.21.15.161:0/3551749635 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rw", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v12) ==== 210+0+167 (secure 0 0 0) 0x7fc0bc157d60 con 0x7fc0bc0d0380 2024-03-23T14:36:21.142 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:36:21.142 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:36:21.142 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow rw" 2024-03-23T14:36:21.142 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-23T14:36:21.143 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-23T14:36:21.143 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-23T14:36:21.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0a006b840 msgr2=0x7fc0a006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0a006b840 0x7fc0a006dd00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc0b8005c00 tx=0x7fc0b8009040 comp rx=0 tx=0).stop 2024-03-23T14:36:21.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0bc0d0380 msgr2=0x7fc0bc148700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc148700 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc0ac00e6e0 tx=0x7fc0ac00eba0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 shutdown_connections 2024-03-23T14:36:21.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0a006b840 0x7fc0a006dd00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0bc1552f0 0x7fc0bc156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.145 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0bc0d0380 0x7fc0bc148700 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.148+0000 7fc0c37c7640 1 --2- 172.21.15.161:0/3551749635 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0bc0b2aa0 0x7fc0bc14a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.149+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 >> 172.21.15.161:0/3551749635 conn(0x7fc0bc06b260 msgr2=0x7fc0bc1544d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:21.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.149+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 shutdown_connections 2024-03-23T14:36:21.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.149+0000 7fc0c37c7640 1 -- 172.21.15.161:0/3551749635 wait complete. 2024-03-23T14:36:21.155 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:21.155 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:21.171 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:21.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.315+0000 7ff063e55640 1 Processor -- start 2024-03-23T14:36:21.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.316+0000 7ff063e55640 1 -- start start 2024-03-23T14:36:21.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.316+0000 7ff063e55640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff05c06eb30 0x7ff05c0b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.316+0000 7ff063e55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 0x7ff05c15b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.316+0000 7ff063e55640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff05c070b90 con 0x7ff05c0b3610 2024-03-23T14:36:21.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.316+0000 7ff063e55640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff05c15ab90 con 0x7ff05c06eb30 2024-03-23T14:36:21.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.316+0000 7ff063e55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff05c0713a0 con 0x7ff05c0b4260 2024-03-23T14:36:21.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.316+0000 7ff061bf1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff05c06eb30 0x7ff05c0b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.317+0000 7ff061bf1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff05c06eb30 0x7ff05c0b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51480/0 (socket says 172.21.15.161:51480) 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.317+0000 7ff061bf1640 1 -- 172.21.15.161:0/787405839 learned_addr learned my addr 172.21.15.161:0/787405839 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.317+0000 7ff0613f0640 1 --2- 172.21.15.161:0/787405839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 0x7ff05c15b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.317+0000 7ff061bf1640 1 -- 172.21.15.161:0/787405839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 msgr2=0x7ff05c15b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.317+0000 7ff061bf1640 1 --2- 172.21.15.161:0/787405839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 0x7ff05c15b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.317+0000 7ff061bf1640 1 -- 172.21.15.161:0/787405839 >> v1:172.21.15.183:6789/0 conn(0x7ff05c0b3610 legacy=0x7ff05c06d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.317+0000 7ff061bf1640 1 -- 172.21.15.161:0/787405839 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05c15b0d0 con 0x7ff05c06eb30 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff061bf1640 1 --2- 172.21.15.161:0/787405839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c0b4870 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff05800aa10 tx=0x7ff05802ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72786525b2912b00 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff060bef640 1 -- 172.21.15.161:0/787405839 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff05802dc10 con 0x7ff05c06eb30 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff060bef640 1 -- 172.21.15.161:0/787405839 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff05802dd70 con 0x7ff05c06eb30 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff060bef640 1 -- 172.21.15.161:0/787405839 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058035660 con 0x7ff05c06eb30 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff063e55640 1 -- 172.21.15.161:0/787405839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 msgr2=0x7ff05c0b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff063e55640 1 --2- 172.21.15.161:0/787405839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c0b4870 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff05800aa10 tx=0x7ff05802ce60 comp rx=0 tx=0).stop 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff063e55640 1 -- 172.21.15.161:0/787405839 shutdown_connections 2024-03-23T14:36:21.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff063e55640 1 --2- 172.21.15.161:0/787405839 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 0x7ff05c15b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff063e55640 1 --2- 172.21.15.161:0/787405839 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c0b4870 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff063e55640 1 -- 172.21.15.161:0/787405839 >> 172.21.15.161:0/787405839 conn(0x7ff05c0b17f0 msgr2=0x7ff05c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:21.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff063e55640 1 -- 172.21.15.161:0/787405839 shutdown_connections 2024-03-23T14:36:21.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.318+0000 7ff063e55640 1 -- 172.21.15.161:0/787405839 wait complete. 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.319+0000 7ff063e55640 1 Processor -- start 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.319+0000 7ff063e55640 1 -- start start 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff063e55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c1f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff063e55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff05c0b3610 0x7ff05c1536d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff063e55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 0x7ff05c0a9c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff063e55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c1f2d20 con 0x7ff05c06eb30 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff063e55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff05c155260 con 0x7ff05c0b3610 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff063e55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff05c1554d0 con 0x7ff05c0b4260 2024-03-23T14:36:21.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff061bf1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c1f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff061bf1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c1f27e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51488/0 (socket says 172.21.15.161:51488) 2024-03-23T14:36:21.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff061bf1640 1 -- 172.21.15.161:0/257251878 learned_addr learned my addr 172.21.15.161:0/257251878 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:21.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff0613f0640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff05c0b3610 0x7ff05c1536d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.320+0000 7ff0623f2640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 0x7ff05c0a9c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff061bf1640 1 -- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 msgr2=0x7ff05c0a9c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff061bf1640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 0x7ff05c0a9c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff061bf1640 1 -- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff05c0b3610 msgr2=0x7ff05c1536d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff061bf1640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff05c0b3610 0x7ff05c1536d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff061bf1640 1 -- 172.21.15.161:0/257251878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff05800a670 con 0x7ff05c06eb30 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff061bf1640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c1f27e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff058000c00 tx=0x7ff058004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff0527fc640 1 -- 172.21.15.161:0/257251878 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058040650 con 0x7ff05c06eb30 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff0527fc640 1 -- 172.21.15.161:0/257251878 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0580407d0 con 0x7ff05c06eb30 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff05c1557b0 con 0x7ff05c06eb30 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff0527fc640 1 -- 172.21.15.161:0/257251878 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff058040b80 con 0x7ff05c06eb30 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.321+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff05c153c10 con 0x7ff05c06eb30 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.322+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff05c155cd0 con 0x7ff05c06eb30 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.322+0000 7ff0527fc640 1 -- 172.21.15.161:0/257251878 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff05803c070 con 0x7ff05c06eb30 2024-03-23T14:36:21.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.323+0000 7ff0527fc640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff04006b810 0x7ff04006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.323+0000 7ff0527fc640 1 -- 172.21.15.161:0/257251878 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7ff0580ac7d0 con 0x7ff05c06eb30 2024-03-23T14:36:21.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.323+0000 7ff0613f0640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff04006b810 0x7ff04006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.323+0000 7ff0613f0640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff04006b810 0x7ff04006dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff05c157e90 tx=0x7ff04c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.327+0000 7ff0527fc640 1 -- 172.21.15.161:0/257251878 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff05c155cd0 con 0x7ff05c06eb30 2024-03-23T14:36:21.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.501+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff05c155cd0 con 0x7ff05c06eb30 2024-03-23T14:36:21.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.503+0000 7ff0527fc640 1 -- 172.21.15.161:0/257251878 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7ff05c155cd0 con 0x7ff05c06eb30 2024-03-23T14:36:21.499 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:21.499 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:21.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff04006b810 msgr2=0x7ff04006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff04006b810 0x7ff04006dcd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff05c157e90 tx=0x7ff04c009190 comp rx=0 tx=0).stop 2024-03-23T14:36:21.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 msgr2=0x7ff05c1f27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c1f27e0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff058000c00 tx=0x7ff058004730 comp rx=0 tx=0).stop 2024-03-23T14:36:21.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 shutdown_connections 2024-03-23T14:36:21.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff04006b810 0x7ff04006dcd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff05c0b4260 0x7ff05c0a9c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff05c0b3610 0x7ff05c1536d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 --2- 172.21.15.161:0/257251878 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff05c06eb30 0x7ff05c1f27e0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 >> 172.21.15.161:0/257251878 conn(0x7ff05c0b17f0 msgr2=0x7ff05c1e5020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:21.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 shutdown_connections 2024-03-23T14:36:21.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.505+0000 7ff063e55640 1 -- 172.21.15.161:0/257251878 wait complete. 2024-03-23T14:36:21.512 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:04.688180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:05.241774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-23T14:36:18.738957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:21.512 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:21.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.619+0000 7ff4e1684640 1 Processor -- start 2024-03-23T14:36:21.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4e1684640 1 -- start start 2024-03-23T14:36:21.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4e1684640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff4dc161820 0x7ff4dc0d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.617 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4e1684640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4dc0d2ba0 0x7ff4dc0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4e1684640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff4dc161c00 con 0x7ff4dc0d1270 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4e1684640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff4dc0cea30 con 0x7ff4dc0d2ba0 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4e1684640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7ff4dc0cfd70 con 0x7ff4dc161820 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4dc0d2ba0 0x7ff4dc0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4db7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4dc0d2ba0 0x7ff4dc0b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51494/0 (socket says 172.21.15.161:51494) 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4da7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff4dc0d1270 0x7ff4dc0d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:49998/0 (socket says 172.21.15.161:49998) 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4db7fe640 1 -- 172.21.15.161:0/3222065433 learned_addr learned my addr 172.21.15.161:0/3222065433 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.620+0000 7ff4daffd640 1 --2- 172.21.15.161:0/3222065433 >> v2:172.21.15.183:3300/0 conn(0x7ff4dc161820 0x7ff4dc0d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.621+0000 7ff4db7fe640 1 -- 172.21.15.161:0/3222065433 >> v2:172.21.15.183:3300/0 conn(0x7ff4dc161820 msgr2=0x7ff4dc0d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.621+0000 7ff4db7fe640 1 --2- 172.21.15.161:0/3222065433 >> v2:172.21.15.183:3300/0 conn(0x7ff4dc161820 0x7ff4dc0d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.621+0000 7ff4db7fe640 1 -- 172.21.15.161:0/3222065433 >> v1:172.21.15.161:6789/0 conn(0x7ff4dc0d1270 legacy=0x7ff4dc0d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.621+0000 7ff4d9ffb640 1 -- 172.21.15.161:0/3222065433 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1541474263 0 0) 0x7ff4dc161c00 con 0x7ff4dc0d1270 2024-03-23T14:36:21.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.621+0000 7ff4db7fe640 1 -- 172.21.15.161:0/3222065433 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4dc0cecd0 con 0x7ff4dc0d2ba0 2024-03-23T14:36:21.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4db7fe640 1 --2- 172.21.15.161:0/3222065433 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc0d2ba0 0x7ff4dc0b0a80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc06b270 tx=0x7ff4d402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fc2691faa1995e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4d9ffb640 1 -- 172.21.15.161:0/3222065433 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4030860 con 0x7ff4dc0d2ba0 2024-03-23T14:36:21.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4d9ffb640 1 -- 172.21.15.161:0/3222065433 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d40309c0 con 0x7ff4dc0d2ba0 2024-03-23T14:36:21.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4d9ffb640 1 -- 172.21.15.161:0/3222065433 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4030d50 con 0x7ff4dc0d2ba0 2024-03-23T14:36:21.621 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4e1684640 1 -- 172.21.15.161:0/3222065433 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc0d2ba0 msgr2=0x7ff4dc0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4e1684640 1 --2- 172.21.15.161:0/3222065433 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc0d2ba0 0x7ff4dc0b0a80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc06b270 tx=0x7ff4d402b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4e1684640 1 -- 172.21.15.161:0/3222065433 shutdown_connections 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4e1684640 1 --2- 172.21.15.161:0/3222065433 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc0d2ba0 0x7ff4dc0b0a80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4e1684640 1 --2- 172.21.15.161:0/3222065433 >> v2:172.21.15.183:3300/0 conn(0x7ff4dc161820 0x7ff4dc0d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4e1684640 1 -- 172.21.15.161:0/3222065433 >> 172.21.15.161:0/3222065433 conn(0x7ff4dc09d190 msgr2=0x7ff4dc0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4e1684640 1 -- 172.21.15.161:0/3222065433 shutdown_connections 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.622+0000 7ff4e1684640 1 -- 172.21.15.161:0/3222065433 wait complete. 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4e1684640 1 Processor -- start 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4e1684640 1 -- start start 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4e1684640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4dc0d1270 0x7ff4dc1f74b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4e1684640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4dc0d2ba0 0x7ff4dc1523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4e1684640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc161820 0x7ff4dc06cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4e1684640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4dc1f8450 con 0x7ff4dc161820 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4e1684640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff4dc0c7c10 con 0x7ff4dc0d2ba0 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4e1684640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff4dc152900 con 0x7ff4dc0d1270 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc161820 0x7ff4dc06cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4db7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc161820 0x7ff4dc06cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51508/0 (socket says 172.21.15.161:51508) 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4db7fe640 1 -- 172.21.15.161:0/1220158076 learned_addr learned my addr 172.21.15.161:0/1220158076 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4da7fc640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4dc0d2ba0 0x7ff4dc1523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.624+0000 7ff4daffd640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4dc0d1270 0x7ff4dc1f74b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4db7fe640 1 -- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4dc0d1270 msgr2=0x7ff4dc1f74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4db7fe640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4dc0d1270 0x7ff4dc1f74b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4db7fe640 1 -- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4dc0d2ba0 msgr2=0x7ff4dc1523c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4db7fe640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4dc0d2ba0 0x7ff4dc1523c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4db7fe640 1 -- 172.21.15.161:0/1220158076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d40094d0 con 0x7ff4dc161820 2024-03-23T14:36:21.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4db7fe640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc161820 0x7ff4dc06cab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff4d402b9e0 tx=0x7ff4d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4c37fe640 1 -- 172.21.15.161:0/1220158076 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4039c90 con 0x7ff4dc161820 2024-03-23T14:36:21.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4c37fe640 1 -- 172.21.15.161:0/1220158076 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4d4039df0 con 0x7ff4dc161820 2024-03-23T14:36:21.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4dc1f7bb0 con 0x7ff4dc161820 2024-03-23T14:36:21.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4c37fe640 1 -- 172.21.15.161:0/1220158076 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4d4043640 con 0x7ff4dc161820 2024-03-23T14:36:21.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.625+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4dc1f8170 con 0x7ff4dc161820 2024-03-23T14:36:21.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.626+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4dc04f4e0 con 0x7ff4dc161820 2024-03-23T14:36:21.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.626+0000 7ff4c37fe640 1 -- 172.21.15.161:0/1220158076 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff4d40437a0 con 0x7ff4dc161820 2024-03-23T14:36:21.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.627+0000 7ff4c37fe640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c806b840 0x7ff4c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.627+0000 7ff4c37fe640 1 -- 172.21.15.161:0/1220158076 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7ff4d403e070 con 0x7ff4dc161820 2024-03-23T14:36:21.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.627+0000 7ff4daffd640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c806b840 0x7ff4c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.627+0000 7ff4daffd640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c806b840 0x7ff4c806dd00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0b2bd0 tx=0x7ff4cc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.631+0000 7ff4c37fe640 1 -- 172.21.15.161:0/1220158076 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff4d4077c40 con 0x7ff4dc161820 2024-03-23T14:36:21.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.802+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ff4dc0d2270 con 0x7ff4dc161820 2024-03-23T14:36:21.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.803+0000 7ff4c37fe640 1 -- 172.21.15.161:0/1220158076 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7ff4d407b300 con 0x7ff4dc161820 2024-03-23T14:36:21.799 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.804+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c806b840 msgr2=0x7ff4c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.804+0000 7ff4e1684640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c806b840 0x7ff4c806dd00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7ff4dc0b2bd0 tx=0x7ff4cc00b040 comp rx=0 tx=0).stop 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.804+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc161820 msgr2=0x7ff4dc06cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.804+0000 7ff4e1684640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc161820 0x7ff4dc06cab0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7ff4d402b9e0 tx=0x7ff4d4004730 comp rx=0 tx=0).stop 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.805+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 shutdown_connections 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.805+0000 7ff4e1684640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4c806b840 0x7ff4c806dd00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.805+0000 7ff4e1684640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4dc161820 0x7ff4dc06cab0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.805+0000 7ff4e1684640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4dc0d2ba0 0x7ff4dc1523c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.805+0000 7ff4e1684640 1 --2- 172.21.15.161:0/1220158076 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4dc0d1270 0x7ff4dc1f74b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.805+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 >> 172.21.15.161:0/1220158076 conn(0x7ff4dc09d190 msgr2=0x7ff4dc15e560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.805+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 shutdown_connections 2024-03-23T14:36:21.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.805+0000 7ff4e1684640 1 -- 172.21.15.161:0/1220158076 wait complete. 2024-03-23T14:36:21.811 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:36:21.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.919+0000 7f1c95fd7640 1 Processor -- start 2024-03-23T14:36:21.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.919+0000 7f1c95fd7640 1 -- start start 2024-03-23T14:36:21.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.919+0000 7f1c95fd7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1c9006e950 0x7f1c9015b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.919+0000 7f1c95fd7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c9015b5b0 0x7f1c9006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c95fd7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c9015ba00 con 0x7f1c901549c0 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c95fd7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1c900b2a10 con 0x7f1c9015b5b0 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c95fd7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f1c90071330 con 0x7f1c9006e950 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c8ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c9015b5b0 0x7f1c9006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c8ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1c9015b5b0 0x7f1c9006d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51516/0 (socket says 172.21.15.161:51516) 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c8ffff640 1 -- 172.21.15.161:0/3968710352 learned_addr learned my addr 172.21.15.161:0/3968710352 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c8effd640 1 --2- 172.21.15.161:0/3968710352 >> v2:172.21.15.183:3300/0 conn(0x7f1c9006e950 0x7f1c9015b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c8ffff640 1 -- 172.21.15.161:0/3968710352 >> v2:172.21.15.183:3300/0 conn(0x7f1c9006e950 msgr2=0x7f1c9015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c8ffff640 1 --2- 172.21.15.161:0/3968710352 >> v2:172.21.15.183:3300/0 conn(0x7f1c9006e950 0x7f1c9015b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.917 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c8ffff640 1 -- 172.21.15.161:0/3968710352 >> v1:172.21.15.183:6789/0 conn(0x7f1c901549c0 legacy=0x7f1c90070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.920+0000 7f1c8ffff640 1 -- 172.21.15.161:0/3968710352 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9006edd0 con 0x7f1c9015b5b0 2024-03-23T14:36:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c8ffff640 1 --2- 172.21.15.161:0/3968710352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c9015b5b0 0x7f1c9006d6f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1c900b27d0 tx=0x7f1c8802bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd33eeaec42a6172 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c8e7fc640 1 -- 172.21.15.161:0/3968710352 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8802cb20 con 0x7f1c9015b5b0 2024-03-23T14:36:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c8e7fc640 1 -- 172.21.15.161:0/3968710352 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8802cc80 con 0x7f1c9015b5b0 2024-03-23T14:36:21.918 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c8e7fc640 1 -- 172.21.15.161:0/3968710352 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c880343f0 con 0x7f1c9015b5b0 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c95fd7640 1 -- 172.21.15.161:0/3968710352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c9015b5b0 msgr2=0x7f1c9006d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/3968710352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c9015b5b0 0x7f1c9006d6f0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f1c900b27d0 tx=0x7f1c8802bf80 comp rx=0 tx=0).stop 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c95fd7640 1 -- 172.21.15.161:0/3968710352 shutdown_connections 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/3968710352 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c9015b5b0 0x7f1c9006d6f0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/3968710352 >> v2:172.21.15.183:3300/0 conn(0x7f1c9006e950 0x7f1c9015b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c95fd7640 1 -- 172.21.15.161:0/3968710352 >> 172.21.15.161:0/3968710352 conn(0x7f1c9006b4a0 msgr2=0x7f1c9006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c95fd7640 1 -- 172.21.15.161:0/3968710352 shutdown_connections 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.921+0000 7f1c95fd7640 1 -- 172.21.15.161:0/3968710352 wait complete. 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.922+0000 7f1c95fd7640 1 Processor -- start 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.923+0000 7f1c95fd7640 1 -- start start 2024-03-23T14:36:21.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.923+0000 7f1c95fd7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c9006e950 0x7f1c9015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.923+0000 7f1c95fd7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c901549c0 0x7f1c90153620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.923+0000 7f1c95fd7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9015b5b0 0x7f1c90153b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.923+0000 7f1c95fd7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1c901531d0 con 0x7f1c901549c0 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.923+0000 7f1c95fd7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1c900ccb30 con 0x7f1c9006e950 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.923+0000 7f1c95fd7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1c900cce10 con 0x7f1c9015b5b0 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.923+0000 7f1c8effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c901549c0 0x7f1c90153620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8f7fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c9006e950 0x7f1c9015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c901549c0 0x7f1c90153620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51524/0 (socket says 172.21.15.161:51524) 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8effd640 1 -- 172.21.15.161:0/2575482388 learned_addr learned my addr 172.21.15.161:0/2575482388 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8ffff640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9015b5b0 0x7f1c90153b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8effd640 1 -- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9015b5b0 msgr2=0x7f1c90153b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8effd640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9015b5b0 0x7f1c90153b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8effd640 1 -- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c9006e950 msgr2=0x7f1c9015cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8effd640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c9006e950 0x7f1c9015cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8effd640 1 -- 172.21.15.161:0/2575482388 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c88009b30 con 0x7f1c901549c0 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8f7fe640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c9006e950 0x7f1c9015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c8effd640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c901549c0 0x7f1c90153620 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400e9b0 tx=0x7f1c8400ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c77fff640 1 -- 172.21.15.161:0/2575482388 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c84009670 con 0x7f1c901549c0 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c77fff640 1 -- 172.21.15.161:0/2575482388 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1c8400c9a0 con 0x7f1c901549c0 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c900cd0f0 con 0x7f1c901549c0 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c77fff640 1 -- 172.21.15.161:0/2575482388 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8400cd50 con 0x7f1c901549c0 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.924+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c901540a0 con 0x7f1c901549c0 2024-03-23T14:36:21.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.925+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c90073570 con 0x7f1c901549c0 2024-03-23T14:36:21.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.926+0000 7f1c77fff640 1 -- 172.21.15.161:0/2575482388 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1c84013470 con 0x7f1c901549c0 2024-03-23T14:36:21.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.926+0000 7f1c77fff640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1c7c06b810 0x7f1c7c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:21.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.926+0000 7f1c77fff640 1 -- 172.21.15.161:0/2575482388 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f1c8408a500 con 0x7f1c901549c0 2024-03-23T14:36:21.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.926+0000 7f1c8f7fe640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1c7c06b810 0x7f1c7c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:21.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.926+0000 7f1c8f7fe640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1c7c06b810 0x7f1c7c06dcd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1c80001b70 tx=0x7f1c80009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:21.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:21.930+0000 7f1c77fff640 1 -- 172.21.15.161:0/2575482388 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1c84054e40 con 0x7f1c901549c0 2024-03-23T14:36:22.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.104+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1c9004f550 con 0x7f1c901549c0 2024-03-23T14:36:22.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.105+0000 7f1c77fff640 1 -- 172.21.15.161:0/2575482388 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) ==== 104+0+11 (secure 0 0 0) 0x7f1c84058500 con 0x7f1c901549c0 2024-03-23T14:36:22.100 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1c7c06b810 msgr2=0x7f1c7c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1c7c06b810 0x7f1c7c06dcd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1c80001b70 tx=0x7f1c80009040 comp rx=0 tx=0).stop 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c901549c0 msgr2=0x7f1c90153620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c901549c0 0x7f1c90153620 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f1c8400e9b0 tx=0x7f1c8400ee70 comp rx=0 tx=0).stop 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 shutdown_connections 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1c7c06b810 0x7f1c7c06dcd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1c9015b5b0 0x7f1c90153b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1c901549c0 0x7f1c90153620 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 --2- 172.21.15.161:0/2575482388 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1c9006e950 0x7f1c9015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 >> 172.21.15.161:0/2575482388 conn(0x7f1c9006b4a0 msgr2=0x7f1c9006b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 shutdown_connections 2024-03-23T14:36:22.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.107+0000 7f1c95fd7640 1 -- 172.21.15.161:0/2575482388 wait complete. 2024-03-23T14:36:22.114 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:36:22.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.222+0000 7fb56db19640 1 Processor -- start 2024-03-23T14:36:22.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.222+0000 7fb56db19640 1 -- start start 2024-03-23T14:36:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.222+0000 7fb56db19640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb568161820 0x7fb5680d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.222+0000 7fb56db19640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.222+0000 7fb56db19640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb568161c00 con 0x7fb5680d29d0 2024-03-23T14:36:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.222+0000 7fb56db19640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fb5680d2e20 con 0x7fb568161820 2024-03-23T14:36:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.222+0000 7fb56db19640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb5680cea30 con 0x7fb5680d1270 2024-03-23T14:36:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb55f7fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb568161820 0x7fb5680d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb5677fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb567fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb5680d29d0 0x7fb568161e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50004/0 (socket says 172.21.15.161:50004) 2024-03-23T14:36:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb567fff640 1 -- 172.21.15.161:0/3582884662 learned_addr learned my addr 172.21.15.161:0/3582884662 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb566ffd640 1 -- 172.21.15.161:0/3582884662 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2149458492 0 0) 0x7fb568161c00 con 0x7fb5680d29d0 2024-03-23T14:36:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb55f7fe640 1 -- 172.21.15.161:0/3582884662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 msgr2=0x7fb5680d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb55f7fe640 1 --2- 172.21.15.161:0/3582884662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb55f7fe640 1 -- 172.21.15.161:0/3582884662 >> v1:172.21.15.161:6789/0 conn(0x7fb5680d29d0 legacy=0x7fb568161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb55f7fe640 1 -- 172.21.15.161:0/3582884662 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5680cfd70 con 0x7fb568161820 2024-03-23T14:36:22.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.223+0000 7fb5677fe640 1 --2- 172.21.15.161:0/3582884662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:22.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb55f7fe640 1 --2- 172.21.15.161:0/3582884662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb568161820 0x7fb5680d47e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb5580097a0 tx=0x7fb55802b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=342394854749f7df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb566ffd640 1 -- 172.21.15.161:0/3582884662 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558031830 con 0x7fb568161820 2024-03-23T14:36:22.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb566ffd640 1 -- 172.21.15.161:0/3582884662 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb558031990 con 0x7fb568161820 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb566ffd640 1 -- 172.21.15.161:0/3582884662 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb558031d20 con 0x7fb568161820 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb56db19640 1 -- 172.21.15.161:0/3582884662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb568161820 msgr2=0x7fb5680d47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb56db19640 1 --2- 172.21.15.161:0/3582884662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb568161820 0x7fb5680d47e0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb5580097a0 tx=0x7fb55802b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb56db19640 1 -- 172.21.15.161:0/3582884662 shutdown_connections 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb56db19640 1 --2- 172.21.15.161:0/3582884662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb56db19640 1 --2- 172.21.15.161:0/3582884662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb568161820 0x7fb5680d47e0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb56db19640 1 -- 172.21.15.161:0/3582884662 >> 172.21.15.161:0/3582884662 conn(0x7fb56809d190 msgr2=0x7fb5680d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.224+0000 7fb56db19640 1 -- 172.21.15.161:0/3582884662 shutdown_connections 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.225+0000 7fb56db19640 1 -- 172.21.15.161:0/3582884662 wait complete. 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.226+0000 7fb56db19640 1 Processor -- start 2024-03-23T14:36:22.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.226+0000 7fb56db19640 1 -- start start 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.226+0000 7fb56db19640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680c94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.226+0000 7fb56db19640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5680d29d0 0x7fb568160a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb56db19640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb568161820 0x7fb5680c8b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb56db19640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb5681f3e70 con 0x7fb568161820 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb56db19640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb5681f79b0 con 0x7fb5680d29d0 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb56db19640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb5681f7c90 con 0x7fb5680d1270 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb567fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb568161820 0x7fb5680c8b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb55f7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680c94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb5677fe640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5680d29d0 0x7fb568160a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb55f7fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680c94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:59282/0 (socket says 172.21.15.161:59282) 2024-03-23T14:36:22.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb55f7fe640 1 -- 172.21.15.161:0/2949161089 learned_addr learned my addr 172.21.15.161:0/2949161089 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb5677fe640 1 -- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 msgr2=0x7fb5680c94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb5677fe640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680c94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb5677fe640 1 -- 172.21.15.161:0/2949161089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb568161820 msgr2=0x7fb5680c8b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb5677fe640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb568161820 0x7fb5680c8b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb5677fe640 1 -- 172.21.15.161:0/2949161089 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb550009400 con 0x7fb5680d29d0 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb567fff640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb568161820 0x7fb5680c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb5677fe640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5680d29d0 0x7fb568160a60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb55000e900 tx=0x7fb55000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb564ff9640 1 -- 172.21.15.161:0/2949161089 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb55000ca00 con 0x7fb5680d29d0 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb564ff9640 1 -- 172.21.15.161:0/2949161089 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb55000cb60 con 0x7fb5680d29d0 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb564ff9640 1 -- 172.21.15.161:0/2949161089 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb550013470 con 0x7fb5680d29d0 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb558009400 con 0x7fb5680d29d0 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.227+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5681f8250 con 0x7fb5680d29d0 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.228+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5681f3fe0 con 0x7fb5680d29d0 2024-03-23T14:36:22.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.229+0000 7fb564ff9640 1 -- 172.21.15.161:0/2949161089 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb5500135d0 con 0x7fb5680d29d0 2024-03-23T14:36:22.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.229+0000 7fb564ff9640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb54806b810 0x7fb54806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.229+0000 7fb564ff9640 1 -- 172.21.15.161:0/2949161089 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fb55008a4d0 con 0x7fb5680d29d0 2024-03-23T14:36:22.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.229+0000 7fb55f7fe640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb54806b810 0x7fb54806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.230+0000 7fb55f7fe640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb54806b810 0x7fb54806dcd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb55802b910 tx=0x7fb55803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.238+0000 7fb564ff9640 1 -- 172.21.15.161:0/2949161089 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb550054e20 con 0x7fb5680d29d0 2024-03-23T14:36:22.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.408+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb56804f4e0 con 0x7fb5680d29d0 2024-03-23T14:36:22.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.409+0000 7fb564ff9640 1 -- 172.21.15.161:0/2949161089 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) ==== 100+0+11 (secure 0 0 0) 0x7fb5500584e0 con 0x7fb5680d29d0 2024-03-23T14:36:22.405 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb54806b810 msgr2=0x7fb54806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb54806b810 0x7fb54806dcd0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fb55802b910 tx=0x7fb55803d040 comp rx=0 tx=0).stop 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5680d29d0 msgr2=0x7fb568160a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5680d29d0 0x7fb568160a60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb55000e900 tx=0x7fb55000edc0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 shutdown_connections 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb54806b810 0x7fb54806dcd0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb568161820 0x7fb5680c8b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5680d29d0 0x7fb568160a60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 --2- 172.21.15.161:0/2949161089 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5680d1270 0x7fb5680c94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 >> 172.21.15.161:0/2949161089 conn(0x7fb56809d190 msgr2=0x7fb5680d0330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:22.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 shutdown_connections 2024-03-23T14:36:22.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.411+0000 7fb56db19640 1 -- 172.21.15.161:0/2949161089 wait complete. 2024-03-23T14:36:22.417 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-23T14:36:22.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.525+0000 7fe838017640 1 Processor -- start 2024-03-23T14:36:22.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.526+0000 7fe838017640 1 -- start start 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.526+0000 7fe838017640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.526+0000 7fe838017640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8300d1330 con 0x7fe8300d2cd0 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.526+0000 7fe838017640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8300d0720 con 0x7fe830161750 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.526+0000 7fe838017640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe8300d0a00 con 0x7fe8300d1cb0 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe8355b2640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe8300d2cd0 0x7fe8300ceb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50014/0 (socket says 172.21.15.161:50014) 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe8355b2640 1 -- 172.21.15.161:0/3128660907 learned_addr learned my addr 172.21.15.161:0/3128660907 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe8365b4640 1 --2- 172.21.15.161:0/3128660907 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe834db1640 1 -- 172.21.15.161:0/3128660907 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1689824970 0 0) 0x7fe8300d1330 con 0x7fe8300d2cd0 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe834db1640 1 -- 172.21.15.161:0/3128660907 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe81c003520 con 0x7fe8300d2cd0 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe834db1640 1 -- 172.21.15.161:0/3128660907 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2024232900 0 0) 0x7fe8300d0720 con 0x7fe830161750 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe834db1640 1 -- 172.21.15.161:0/3128660907 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8300d1330 con 0x7fe830161750 2024-03-23T14:36:22.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe8365b4640 1 -- 172.21.15.161:0/3128660907 >> v1:172.21.15.183:6789/0 conn(0x7fe830161750 legacy=0x7fe8300d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe8365b4640 1 -- 172.21.15.161:0/3128660907 >> v1:172.21.15.161:6789/0 conn(0x7fe8300d2cd0 legacy=0x7fe8300ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.527+0000 7fe8365b4640 1 -- 172.21.15.161:0/3128660907 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8300d0ce0 con 0x7fe8300d1cb0 2024-03-23T14:36:22.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe8365b4640 1 --2- 172.21.15.161:0/3128660907 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300cf220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe828009dc0 tx=0x7fe82802bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab9254e2f9bf6fcc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe834db1640 1 -- 172.21.15.161:0/3128660907 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82802ccd0 con 0x7fe8300d1cb0 2024-03-23T14:36:22.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe834db1640 1 -- 172.21.15.161:0/3128660907 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82802ce30 con 0x7fe8300d1cb0 2024-03-23T14:36:22.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe834db1640 1 -- 172.21.15.161:0/3128660907 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe828034640 con 0x7fe8300d1cb0 2024-03-23T14:36:22.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe838017640 1 -- 172.21.15.161:0/3128660907 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 msgr2=0x7fe8300cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe838017640 1 --2- 172.21.15.161:0/3128660907 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300cf220 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe828009dc0 tx=0x7fe82802bf20 comp rx=0 tx=0).stop 2024-03-23T14:36:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe838017640 1 -- 172.21.15.161:0/3128660907 shutdown_connections 2024-03-23T14:36:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe838017640 1 --2- 172.21.15.161:0/3128660907 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300cf220 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe838017640 1 -- 172.21.15.161:0/3128660907 >> 172.21.15.161:0/3128660907 conn(0x7fe83009d450 msgr2=0x7fe8300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe838017640 1 -- 172.21.15.161:0/3128660907 shutdown_connections 2024-03-23T14:36:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.528+0000 7fe838017640 1 -- 172.21.15.161:0/3128660907 wait complete. 2024-03-23T14:36:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe838017640 1 Processor -- start 2024-03-23T14:36:22.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe838017640 1 -- start start 2024-03-23T14:36:22.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe838017640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300b9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe838017640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8300d2cd0 0x7fe8300b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe838017640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe830161750 0x7fe8300cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe838017640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe8300b7520 con 0x7fe830161750 2024-03-23T14:36:22.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe838017640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe8300ca1b0 con 0x7fe8300d2cd0 2024-03-23T14:36:22.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe838017640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe8300c9810 con 0x7fe8300d1cb0 2024-03-23T14:36:22.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe8365b4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe830161750 0x7fe8300cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe8365b4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe830161750 0x7fe8300cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51538/0 (socket says 172.21.15.161:51538) 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe8365b4640 1 -- 172.21.15.161:0/2494365070 learned_addr learned my addr 172.21.15.161:0/2494365070 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.530+0000 7fe835db3640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300b9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8355b2640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8300d2cd0 0x7fe8300b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8365b4640 1 -- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 msgr2=0x7fe8300b9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8365b4640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300b9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8365b4640 1 -- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8300d2cd0 msgr2=0x7fe8300b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8365b4640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8300d2cd0 0x7fe8300b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8365b4640 1 -- 172.21.15.161:0/2494365070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe828009a70 con 0x7fe830161750 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8365b4640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe830161750 0x7fe8300cd980 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe828009ee0 tx=0x7fe828004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8267fc640 1 -- 172.21.15.161:0/2494365070 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82803d600 con 0x7fe830161750 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8267fc640 1 -- 172.21.15.161:0/2494365070 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe82803d760 con 0x7fe830161750 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8300c9af0 con 0x7fe830161750 2024-03-23T14:36:22.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe8267fc640 1 -- 172.21.15.161:0/2494365070 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe82803db10 con 0x7fe830161750 2024-03-23T14:36:22.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.531+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8300b7210 con 0x7fe830161750 2024-03-23T14:36:22.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.532+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8300ca2f0 con 0x7fe830161750 2024-03-23T14:36:22.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.532+0000 7fe8267fc640 1 -- 172.21.15.161:0/2494365070 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe82803b040 con 0x7fe830161750 2024-03-23T14:36:22.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.533+0000 7fe8267fc640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe81406b810 0x7fe81406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.533+0000 7fe835db3640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe81406b810 0x7fe81406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.533+0000 7fe8267fc640 1 -- 172.21.15.161:0/2494365070 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fe8280ac700 con 0x7fe830161750 2024-03-23T14:36:22.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.534+0000 7fe835db3640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe81406b810 0x7fe81406dcd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe82c007fd0 tx=0x7fe82c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.533 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.537+0000 7fe8267fc640 1 -- 172.21.15.161:0/2494365070 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe8300ca2f0 con 0x7fe830161750 2024-03-23T14:36:22.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.707+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fe8300ca2f0 con 0x7fe830161750 2024-03-23T14:36:22.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.707+0000 7fe8267fc640 1 -- 172.21.15.161:0/2494365070 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v36) ==== 97+0+11 (secure 0 0 0) 0x7fe8300ca2f0 con 0x7fe830161750 2024-03-23T14:36:22.704 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:22.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe81406b810 msgr2=0x7fe81406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.705 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe81406b810 0x7fe81406dcd0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fe82c007fd0 tx=0x7fe82c009040 comp rx=0 tx=0).stop 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe830161750 msgr2=0x7fe8300cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe830161750 0x7fe8300cd980 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fe828009ee0 tx=0x7fe828004a40 comp rx=0 tx=0).stop 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 shutdown_connections 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe81406b810 0x7fe81406dcd0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe830161750 0x7fe8300cd980 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe8300d2cd0 0x7fe8300b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 --2- 172.21.15.161:0/2494365070 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe8300d1cb0 0x7fe8300b9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.709+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 >> 172.21.15.161:0/2494365070 conn(0x7fe83009d450 msgr2=0x7fe83006bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.710+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 shutdown_connections 2024-03-23T14:36:22.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.710+0000 7fe838017640 1 -- 172.21.15.161:0/2494365070 wait complete. 2024-03-23T14:36:22.716 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:36:22.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.821+0000 7f2d14695640 1 Processor -- start 2024-03-23T14:36:22.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.821+0000 7f2d14695640 1 -- start start 2024-03-23T14:36:22.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d14695640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2d0c159b50 0x7f2d0c15cf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d14695640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2d0c0d0320 0x7f2d0c0ce430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d14695640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d0c159f30 con 0x7f2d0c071ac0 2024-03-23T14:36:22.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d14695640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0c071630 con 0x7f2d0c0d0320 2024-03-23T14:36:22.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d14695640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f2d0c15d470 con 0x7f2d0c159b50 2024-03-23T14:36:22.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d12c32640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2d0c0d0320 0x7f2d0c0ce430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d11c30640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2d0c071ac0 0x7f2d0c0cfc10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50018/0 (socket says 172.21.15.161:50018) 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d11c30640 1 -- 172.21.15.161:0/2697960110 learned_addr learned my addr 172.21.15.161:0/2697960110 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d12431640 1 --2- 172.21.15.161:0/2697960110 >> v2:172.21.15.183:3300/0 conn(0x7f2d0c159b50 0x7f2d0c15cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d12c32640 1 -- 172.21.15.161:0/2697960110 >> v2:172.21.15.183:3300/0 conn(0x7f2d0c159b50 msgr2=0x7f2d0c15cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d12c32640 1 --2- 172.21.15.161:0/2697960110 >> v2:172.21.15.183:3300/0 conn(0x7f2d0c159b50 0x7f2d0c15cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d12c32640 1 -- 172.21.15.161:0/2697960110 >> v1:172.21.15.161:6789/0 conn(0x7f2d0c071ac0 legacy=0x7f2d0c0cfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d12c32640 1 -- 172.21.15.161:0/2697960110 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d0c071020 con 0x7f2d0c0d0320 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.822+0000 7f2d1142f640 1 -- 172.21.15.161:0/2697960110 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2651943646 0 0) 0x7f2cfc001110 con 0x7f2d0c071ac0 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.823+0000 7f2d12c32640 1 --2- 172.21.15.161:0/2697960110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c0ce430 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2d04009d90 tx=0x7f2d0402be20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a958b377577c804b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.827 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.823+0000 7f2d1142f640 1 -- 172.21.15.161:0/2697960110 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0402cbd0 con 0x7f2d0c0d0320 2024-03-23T14:36:22.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.823+0000 7f2d1142f640 1 -- 172.21.15.161:0/2697960110 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d0402cd30 con 0x7f2d0c0d0320 2024-03-23T14:36:22.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.823+0000 7f2d1142f640 1 -- 172.21.15.161:0/2697960110 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d0402cea0 con 0x7f2d0c0d0320 2024-03-23T14:36:22.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.824+0000 7f2d14695640 1 -- 172.21.15.161:0/2697960110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 msgr2=0x7f2d0c0ce430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.824+0000 7f2d14695640 1 --2- 172.21.15.161:0/2697960110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c0ce430 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f2d04009d90 tx=0x7f2d0402be20 comp rx=0 tx=0).stop 2024-03-23T14:36:22.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.824+0000 7f2d14695640 1 -- 172.21.15.161:0/2697960110 shutdown_connections 2024-03-23T14:36:22.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.824+0000 7f2d14695640 1 --2- 172.21.15.161:0/2697960110 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c0ce430 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.824+0000 7f2d14695640 1 --2- 172.21.15.161:0/2697960110 >> v2:172.21.15.183:3300/0 conn(0x7f2d0c159b50 0x7f2d0c15cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.828 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.824+0000 7f2d14695640 1 -- 172.21.15.161:0/2697960110 >> 172.21.15.161:0/2697960110 conn(0x7f2d0c06d0e0 msgr2=0x7f2d0c06d4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.824+0000 7f2d14695640 1 -- 172.21.15.161:0/2697960110 shutdown_connections 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.824+0000 7f2d14695640 1 -- 172.21.15.161:0/2697960110 wait complete. 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.825+0000 7f2d14695640 1 Processor -- start 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.825+0000 7f2d14695640 1 -- start start 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d14695640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d0c071ac0 0x7f2d0c1552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d14695640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c1e79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d14695640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d0c159b50 0x7f2d0c1e7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d14695640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c1e6a50 con 0x7f2d0c0d0320 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d14695640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2d0c1e5e40 con 0x7f2d0c159b50 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d14695640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2d0c1e6120 con 0x7f2d0c071ac0 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c1e79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c1e79a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51548/0 (socket says 172.21.15.161:51548) 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 -- 172.21.15.161:0/2880399125 learned_addr learned my addr 172.21.15.161:0/2880399125 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d12431640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d0c071ac0 0x7f2d0c1552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.829 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d12c32640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d0c159b50 0x7f2d0c1e7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 -- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d0c071ac0 msgr2=0x7f2d0c1552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d0c071ac0 0x7f2d0c1552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 -- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d0c159b50 msgr2=0x7f2d0c1e7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d0c159b50 0x7f2d0c1e7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 -- 172.21.15.161:0/2880399125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d04009a70 con 0x7f2d0c0d0320 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d11c30640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c1e79a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c780 tx=0x7f2cfc00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d02ffd640 1 -- 172.21.15.161:0/2880399125 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc0072d0 con 0x7f2d0c0d0320 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d02ffd640 1 -- 172.21.15.161:0/2880399125 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cfc007430 con 0x7f2d0c0d0320 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d0c1e6400 con 0x7f2d0c0d0320 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d02ffd640 1 -- 172.21.15.161:0/2880399125 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cfc001d10 con 0x7f2d0c0d0320 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.826+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d0c155820 con 0x7f2d0c0d0320 2024-03-23T14:36:22.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.827+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d0c1f1ce0 con 0x7f2d0c0d0320 2024-03-23T14:36:22.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.828+0000 7f2d02ffd640 1 -- 172.21.15.161:0/2880399125 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2cfc0047e0 con 0x7f2d0c0d0320 2024-03-23T14:36:22.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.828+0000 7f2d02ffd640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2cf006b840 0x7f2cf006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:22.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.828+0000 7f2d02ffd640 1 -- 172.21.15.161:0/2880399125 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f2cfc0895e0 con 0x7f2d0c0d0320 2024-03-23T14:36:22.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.828+0000 7f2d12431640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2cf006b840 0x7f2cf006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:22.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.829+0000 7f2d12431640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2cf006b840 0x7f2cf006dd00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c15bde0 tx=0x7f2d08009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:22.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:22.833+0000 7f2d02ffd640 1 -- 172.21.15.161:0/2880399125 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2d0c1f1ce0 con 0x7f2d0c0d0320 2024-03-23T14:36:23.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.020+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2d0c159f30 con 0x7f2d0c0d0320 2024-03-23T14:36:23.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.021+0000 7f2d02ffd640 1 -- 172.21.15.161:0/2880399125 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) ==== 80+0+0 (secure 0 0 0) 0x7f2d0c159f30 con 0x7f2d0c0d0320 2024-03-23T14:36:23.018 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:36:23.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2cf006b840 msgr2=0x7f2cf006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2cf006b840 0x7f2cf006dd00 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f2d0c15bde0 tx=0x7f2d08009190 comp rx=0 tx=0).stop 2024-03-23T14:36:23.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 msgr2=0x7f2d0c1e79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c1e79a0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2cfc00c780 tx=0x7f2cfc00cc40 comp rx=0 tx=0).stop 2024-03-23T14:36:23.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 shutdown_connections 2024-03-23T14:36:23.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2cf006b840 0x7f2cf006dd00 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.020 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2d0c159b50 0x7f2d0c1e7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2d0c0d0320 0x7f2d0c1e79a0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 --2- 172.21.15.161:0/2880399125 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2d0c071ac0 0x7f2d0c1552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 >> 172.21.15.161:0/2880399125 conn(0x7f2d0c06d0e0 msgr2=0x7f2d0c0ce3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:23.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 shutdown_connections 2024-03-23T14:36:23.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.023+0000 7f2d14695640 1 -- 172.21.15.161:0/2880399125 wait complete. 2024-03-23T14:36:23.030 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:23.030 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:23.047 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:23.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.201+0000 7f9ad1a2e640 1 Processor -- start 2024-03-23T14:36:23.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.201+0000 7f9ad1a2e640 1 -- start start 2024-03-23T14:36:23.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9ad1a2e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9acc0cdf10 0x7f9acc06f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9ad1a2e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9acc0d0f80 0x7f9acc0ce580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.207 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9ad1a2e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9acc0701e0 con 0x7f9acc0d2820 2024-03-23T14:36:23.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9ad1a2e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9acc0d0af0 con 0x7f9acc0d0f80 2024-03-23T14:36:23.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9ad1a2e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f9acc0cfee0 con 0x7f9acc0cdf10 2024-03-23T14:36:23.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9aca7fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9acc0d0f80 0x7f9acc0ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9acb7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9acc0d2820 0x7f9acc161dc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50024/0 (socket says 172.21.15.161:50024) 2024-03-23T14:36:23.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9acb7fe640 1 -- 172.21.15.161:0/3162850724 learned_addr learned my addr 172.21.15.161:0/3162850724 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:23.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9acaffd640 1 --2- 172.21.15.161:0/3162850724 >> v2:172.21.15.183:3300/0 conn(0x7f9acc0cdf10 0x7f9acc06f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.208 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9aca7fc640 1 -- 172.21.15.161:0/3162850724 >> v2:172.21.15.183:3300/0 conn(0x7f9acc0cdf10 msgr2=0x7f9acc06f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:23.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9aca7fc640 1 --2- 172.21.15.161:0/3162850724 >> v2:172.21.15.183:3300/0 conn(0x7f9acc0cdf10 0x7f9acc06f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9aca7fc640 1 -- 172.21.15.161:0/3162850724 >> v1:172.21.15.161:6789/0 conn(0x7f9acc0d2820 legacy=0x7f9acc161dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9aca7fc640 1 -- 172.21.15.161:0/3162850724 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9acc0d01c0 con 0x7f9acc0d0f80 2024-03-23T14:36:23.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.202+0000 7f9ac9ffb640 1 -- 172.21.15.161:0/3162850724 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3733162886 0 0) 0x7f9acc0701e0 con 0x7f9acc0d2820 2024-03-23T14:36:23.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.203+0000 7f9aca7fc640 1 --2- 172.21.15.161:0/3162850724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d0f80 0x7f9acc0ce580 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9acc0cf800 tx=0x7f9ac002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30a73a4ca32cb4d1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.209 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.203+0000 7f9ac9ffb640 1 -- 172.21.15.161:0/3162850724 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac002e070 con 0x7f9acc0d0f80 2024-03-23T14:36:23.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.203+0000 7f9ac9ffb640 1 -- 172.21.15.161:0/3162850724 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac00315a0 con 0x7f9acc0d0f80 2024-03-23T14:36:23.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.203+0000 7f9ac9ffb640 1 -- 172.21.15.161:0/3162850724 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac0031930 con 0x7f9acc0d0f80 2024-03-23T14:36:23.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.204+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3162850724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d0f80 msgr2=0x7f9acc0ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.204+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3162850724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d0f80 0x7f9acc0ce580 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f9acc0cf800 tx=0x7f9ac002b840 comp rx=0 tx=0).stop 2024-03-23T14:36:23.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.204+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3162850724 shutdown_connections 2024-03-23T14:36:23.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.204+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3162850724 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d0f80 0x7f9acc0ce580 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.210 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.204+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3162850724 >> v2:172.21.15.183:3300/0 conn(0x7f9acc0cdf10 0x7f9acc06f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.204+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3162850724 >> 172.21.15.161:0/3162850724 conn(0x7f9acc0d1d00 msgr2=0x7f9acc0d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:23.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.204+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3162850724 shutdown_connections 2024-03-23T14:36:23.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.204+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3162850724 wait complete. 2024-03-23T14:36:23.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.205+0000 7f9ad1a2e640 1 Processor -- start 2024-03-23T14:36:23.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.205+0000 7f9ad1a2e640 1 -- start start 2024-03-23T14:36:23.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ad1a2e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9acc0cdf10 0x7f9acc0c8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ad1a2e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9acc0d0f80 0x7f9acc152ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ad1a2e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d2820 0x7f9acc0cc600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ad1a2e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9acc09dd50 con 0x7f9acc0d2820 2024-03-23T14:36:23.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ad1a2e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9acc153530 con 0x7f9acc0cdf10 2024-03-23T14:36:23.212 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ad1a2e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9acc09dff0 con 0x7f9acc0d0f80 2024-03-23T14:36:23.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d2820 0x7f9acc0cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d2820 0x7f9acc0cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51560/0 (socket says 172.21.15.161:51560) 2024-03-23T14:36:23.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acaffd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9acc0cdf10 0x7f9acc0c8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 -- 172.21.15.161:0/3676749049 learned_addr learned my addr 172.21.15.161:0/3676749049 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:23.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9aca7fc640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9acc0d0f80 0x7f9acc152ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 -- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9acc0d0f80 msgr2=0x7f9acc152ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9acc0d0f80 0x7f9acc152ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 -- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9acc0cdf10 msgr2=0x7f9acc0c8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9acc0cdf10 0x7f9acc0c8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 -- 172.21.15.161:0/3676749049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac0009400 con 0x7f9acc0d2820 2024-03-23T14:36:23.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9acb7fe640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d2820 0x7f9acc0cc600 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400c780 tx=0x7f9ac400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ab3fff640 1 -- 172.21.15.161:0/3676749049 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac400ce90 con 0x7f9acc0d2820 2024-03-23T14:36:23.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ab3fff640 1 -- 172.21.15.161:0/3676749049 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ac400a640 con 0x7f9acc0d2820 2024-03-23T14:36:23.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9acc09e2d0 con 0x7f9acc0d2820 2024-03-23T14:36:23.214 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.206+0000 7f9ab3fff640 1 -- 172.21.15.161:0/3676749049 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ac400a9d0 con 0x7f9acc0d2820 2024-03-23T14:36:23.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.207+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9acc09e890 con 0x7f9acc0d2820 2024-03-23T14:36:23.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.207+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9acc09ea80 con 0x7f9acc0d2820 2024-03-23T14:36:23.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.208+0000 7f9ab3fff640 1 -- 172.21.15.161:0/3676749049 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9ac400ab30 con 0x7f9acc0d2820 2024-03-23T14:36:23.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.208+0000 7f9ab3fff640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9ab806b810 0x7f9ab806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.208+0000 7f9ab3fff640 1 -- 172.21.15.161:0/3676749049 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f9ac4089a70 con 0x7f9acc0d2820 2024-03-23T14:36:23.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.209+0000 7f9acaffd640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9ab806b810 0x7f9ab806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.209+0000 7f9acaffd640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9ab806b810 0x7f9ab806dcd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009b40 tx=0x7f9abc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.213+0000 7f9ab3fff640 1 -- 172.21.15.161:0/3676749049 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9ac4054440 con 0x7f9acc0d2820 2024-03-23T14:36:23.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.391+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9acc04f550 con 0x7f9acc0d2820 2024-03-23T14:36:23.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.392+0000 7f9ab3fff640 1 -- 172.21.15.161:0/3676749049 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7f9ac4018020 con 0x7f9acc0d2820 2024-03-23T14:36:23.391 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:23.391 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:23.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9ab806b810 msgr2=0x7f9ab806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9ab806b810 0x7f9ab806dcd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f9abc009b40 tx=0x7f9abc009190 comp rx=0 tx=0).stop 2024-03-23T14:36:23.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d2820 msgr2=0x7f9acc0cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d2820 0x7f9acc0cc600 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f9ac400c780 tx=0x7f9ac400cc40 comp rx=0 tx=0).stop 2024-03-23T14:36:23.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 shutdown_connections 2024-03-23T14:36:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9ab806b810 0x7f9ab806dcd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9acc0d2820 0x7f9acc0cc600 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9acc0d0f80 0x7f9acc152ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 --2- 172.21.15.161:0/3676749049 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9acc0cdf10 0x7f9acc0c8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.394+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 >> 172.21.15.161:0/3676749049 conn(0x7f9acc0d1d00 msgr2=0x7f9acc0af930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.395+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 shutdown_connections 2024-03-23T14:36:23.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.395+0000 7f9ad1a2e640 1 -- 172.21.15.161:0/3676749049 wait complete. 2024-03-23T14:36:23.404 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:04.688180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:05.241774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-23T14:36:18.738957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:23.405 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:23.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.516+0000 7f32e11e9640 1 Processor -- start 2024-03-23T14:36:23.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.516+0000 7f32e11e9640 1 -- start start 2024-03-23T14:36:23.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32e11e9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32dc145570 0x7f32dc0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32e11e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc15cdf0 0x7f32dc0cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32e11e9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32dc0cf200 con 0x7f32dc0d01b0 2024-03-23T14:36:23.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32e11e9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f32dc0d05f0 con 0x7f32dc145570 2024-03-23T14:36:23.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32e11e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f32dc15d270 con 0x7f32dc15cdf0 2024-03-23T14:36:23.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32dad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32dc145570 0x7f32dc0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32dad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f32dc145570 0x7f32dc0cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51572/0 (socket says 172.21.15.161:51572) 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32dad9d640 1 -- 172.21.15.161:0/1416753862 learned_addr learned my addr 172.21.15.161:0/1416753862 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32da59c640 1 --2- 172.21.15.161:0/1416753862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc15cdf0 0x7f32dc0cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32dad9d640 1 -- 172.21.15.161:0/1416753862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc15cdf0 msgr2=0x7f32dc0cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32dad9d640 1 --2- 172.21.15.161:0/1416753862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc15cdf0 0x7f32dc0cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32dad9d640 1 -- 172.21.15.161:0/1416753862 >> v1:172.21.15.183:6789/0 conn(0x7f32dc0d01b0 legacy=0x7f32dc070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32dad9d640 1 -- 172.21.15.161:0/1416753862 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32dc0ce5f0 con 0x7f32dc145570 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.517+0000 7f32d9d9b640 1 -- 172.21.15.161:0/1416753862 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2860356049 0 0) 0x7f32dc0cf200 con 0x7f32dc0d01b0 2024-03-23T14:36:23.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.518+0000 7f32dad9d640 1 --2- 172.21.15.161:0/1416753862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc145570 0x7f32dc0cdf30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009e50 tx=0x7f32cc02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0be71ae0b3caf60 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.518+0000 7f32d9d9b640 1 -- 172.21.15.161:0/1416753862 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32cc02cb00 con 0x7f32dc145570 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.518+0000 7f32d9d9b640 1 -- 172.21.15.161:0/1416753862 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32cc02cc60 con 0x7f32dc145570 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.518+0000 7f32d9d9b640 1 -- 172.21.15.161:0/1416753862 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32cc0343f0 con 0x7f32dc145570 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.519+0000 7f32e11e9640 1 -- 172.21.15.161:0/1416753862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc145570 msgr2=0x7f32dc0cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.519+0000 7f32e11e9640 1 --2- 172.21.15.161:0/1416753862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc145570 0x7f32dc0cdf30 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f32cc009e50 tx=0x7f32cc02bf60 comp rx=0 tx=0).stop 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.519+0000 7f32e11e9640 1 -- 172.21.15.161:0/1416753862 shutdown_connections 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.519+0000 7f32e11e9640 1 --2- 172.21.15.161:0/1416753862 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc15cdf0 0x7f32dc0cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.519+0000 7f32e11e9640 1 --2- 172.21.15.161:0/1416753862 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc145570 0x7f32dc0cdf30 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.519+0000 7f32e11e9640 1 -- 172.21.15.161:0/1416753862 >> 172.21.15.161:0/1416753862 conn(0x7f32dc06b260 msgr2=0x7f32dc06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:23.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.519+0000 7f32e11e9640 1 -- 172.21.15.161:0/1416753862 shutdown_connections 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.519+0000 7f32e11e9640 1 -- 172.21.15.161:0/1416753862 wait complete. 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.520+0000 7f32e11e9640 1 Processor -- start 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.520+0000 7f32e11e9640 1 -- start start 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32e11e9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc0d01b0 0x7f32dc158840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32e11e9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32dc145570 0x7f32dc12f830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32e11e9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc15cdf0 0x7f32dc0aa910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32e11e9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f32dc0aa070 con 0x7f32dc15cdf0 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32e11e9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f32dc0a94a0 con 0x7f32dc145570 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32e11e9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f32dc0a9740 con 0x7f32dc0d01b0 2024-03-23T14:36:23.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc15cdf0 0x7f32dc0aa910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc15cdf0 0x7f32dc0aa910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51586/0 (socket says 172.21.15.161:51586) 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 -- 172.21.15.161:0/2330507581 learned_addr learned my addr 172.21.15.161:0/2330507581 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32da59c640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32dc145570 0x7f32dc12f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32dad9d640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc0d01b0 0x7f32dc158840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 -- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc0d01b0 msgr2=0x7f32dc158840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc0d01b0 0x7f32dc158840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 -- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32dc145570 msgr2=0x7f32dc12f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32dc145570 0x7f32dc12f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 -- 172.21.15.161:0/2330507581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32cc009b30 con 0x7f32dc15cdf0 2024-03-23T14:36:23.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.521+0000 7f32db59e640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc15cdf0 0x7f32dc0aa910 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f32d400c390 tx=0x7f32d400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.522+0000 7f32c37fe640 1 -- 172.21.15.161:0/2330507581 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d40035f0 con 0x7f32dc15cdf0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.522+0000 7f32c37fe640 1 -- 172.21.15.161:0/2330507581 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f32d4003750 con 0x7f32dc15cdf0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.522+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32dc0a9a20 con 0x7f32dc15cdf0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.522+0000 7f32c37fe640 1 -- 172.21.15.161:0/2330507581 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d4003b00 con 0x7f32dc15cdf0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.522+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32dc158d80 con 0x7f32dc15cdf0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.523+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32dc0aa220 con 0x7f32dc15cdf0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.523+0000 7f32c37fe640 1 -- 172.21.15.161:0/2330507581 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f32d4003c60 con 0x7f32dc15cdf0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.523+0000 7f32c37fe640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f32c806b840 0x7f32c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.523+0000 7f32c37fe640 1 -- 172.21.15.161:0/2330507581 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f32d40897c0 con 0x7f32dc15cdf0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.524+0000 7f32dad9d640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f32c806b840 0x7f32c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.524+0000 7f32dad9d640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f32c806b840 0x7f32c806dd00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f32cc035000 tx=0x7f32cc030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.528+0000 7f32c37fe640 1 -- 172.21.15.161:0/2330507581 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f32dc0aa220 con 0x7f32dc15cdf0 2024-03-23T14:36:23.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.698+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f32dc04f4e0 con 0x7f32dc15cdf0 2024-03-23T14:36:23.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.699+0000 7f32c37fe640 1 -- 172.21.15.161:0/2330507581 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f32d4054190 con 0x7f32dc15cdf0 2024-03-23T14:36:23.695 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f32c806b840 msgr2=0x7f32c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f32c806b840 0x7f32c806dd00 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f32cc035000 tx=0x7f32cc030000 comp rx=0 tx=0).stop 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc15cdf0 msgr2=0x7f32dc0aa910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc15cdf0 0x7f32dc0aa910 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f32d400c390 tx=0x7f32d400c850 comp rx=0 tx=0).stop 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 shutdown_connections 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f32c806b840 0x7f32c806dd00 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f32dc15cdf0 0x7f32dc0aa910 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f32dc145570 0x7f32dc12f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 --2- 172.21.15.161:0/2330507581 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f32dc0d01b0 0x7f32dc158840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 >> 172.21.15.161:0/2330507581 conn(0x7f32dc06b260 msgr2=0x7f32dc0b33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:23.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 shutdown_connections 2024-03-23T14:36:23.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.701+0000 7f32e11e9640 1 -- 172.21.15.161:0/2330507581 wait complete. 2024-03-23T14:36:23.708 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:36:23.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.816+0000 7f9b35a6b640 1 Processor -- start 2024-03-23T14:36:23.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.816+0000 7f9b35a6b640 1 -- start start 2024-03-23T14:36:23.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.817+0000 7f9b35a6b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9b30161750 0x7f9b300d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.817+0000 7f9b35a6b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b300d1330 con 0x7f9b30161b30 2024-03-23T14:36:23.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.817+0000 7f9b35a6b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9b30162e80 con 0x7f9b300d1cb0 2024-03-23T14:36:23.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.817+0000 7f9b35a6b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9b300d0720 con 0x7f9b30161750 2024-03-23T14:36:23.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.817+0000 7f9b2f7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9b30161750 0x7f9b300d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.816 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.817+0000 7f9b2ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9b30161b30 0x7f9b30162770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50032/0 (socket says 172.21.15.161:50032) 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.817+0000 7f9b2ffff640 1 -- 172.21.15.161:0/911207510 learned_addr learned my addr 172.21.15.161:0/911207510 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.818+0000 7f9b2f7fe640 1 -- 172.21.15.161:0/911207510 >> v1:172.21.15.183:6789/0 conn(0x7f9b300d1cb0 legacy=0x7f9b30162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.818+0000 7f9b2effd640 1 -- 172.21.15.161:0/911207510 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1158525034 0 0) 0x7f9b30162e80 con 0x7f9b300d1cb0 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.818+0000 7f9b2f7fe640 1 -- 172.21.15.161:0/911207510 >> v1:172.21.15.161:6789/0 conn(0x7f9b30161b30 legacy=0x7f9b30162770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.818+0000 7f9b2f7fe640 1 -- 172.21.15.161:0/911207510 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b300d0a00 con 0x7f9b30161750 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.818+0000 7f9b2f7fe640 1 --2- 172.21.15.161:0/911207510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b30161750 0x7f9b300d25c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9b200097a0 tx=0x7f9b2002b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48e5562ced654161 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b2effd640 1 -- 172.21.15.161:0/911207510 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b20030710 con 0x7f9b30161750 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b2effd640 1 -- 172.21.15.161:0/911207510 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b2002e050 con 0x7f9b30161750 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b2effd640 1 -- 172.21.15.161:0/911207510 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b20030a90 con 0x7f9b30161750 2024-03-23T14:36:23.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b35a6b640 1 -- 172.21.15.161:0/911207510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b30161750 msgr2=0x7f9b300d25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b35a6b640 1 --2- 172.21.15.161:0/911207510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b30161750 0x7f9b300d25c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f9b200097a0 tx=0x7f9b2002b790 comp rx=0 tx=0).stop 2024-03-23T14:36:23.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b35a6b640 1 -- 172.21.15.161:0/911207510 shutdown_connections 2024-03-23T14:36:23.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b35a6b640 1 --2- 172.21.15.161:0/911207510 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b30161750 0x7f9b300d25c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b35a6b640 1 -- 172.21.15.161:0/911207510 >> 172.21.15.161:0/911207510 conn(0x7f9b3009d450 msgr2=0x7f9b300d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:23.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b35a6b640 1 -- 172.21.15.161:0/911207510 shutdown_connections 2024-03-23T14:36:23.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.819+0000 7f9b35a6b640 1 -- 172.21.15.161:0/911207510 wait complete. 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.820+0000 7f9b35a6b640 1 Processor -- start 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b35a6b640 1 -- start start 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b35a6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b300d1cb0 0x7f9b300cdd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b35a6b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b30161750 0x7f9b30158f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b35a6b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b30161b30 0x7f9b300a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b35a6b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9b300cc7c0 con 0x7f9b300d1cb0 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b35a6b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9b3015d140 con 0x7f9b30161750 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b35a6b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9b3015d420 con 0x7f9b30161b30 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b300d1cb0 0x7f9b300cdd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b2f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b300d1cb0 0x7f9b300cdd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51596/0 (socket says 172.21.15.161:51596) 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b2f7fe640 1 -- 172.21.15.161:0/767112889 learned_addr learned my addr 172.21.15.161:0/767112889 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:23.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b26ffd640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b30161750 0x7f9b30158f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.821+0000 7f9b2ffff640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b30161b30 0x7f9b300a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2f7fe640 1 -- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b30161b30 msgr2=0x7f9b300a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2f7fe640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b30161b30 0x7f9b300a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2f7fe640 1 -- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b30161750 msgr2=0x7f9b30158f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2f7fe640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b30161750 0x7f9b30158f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2f7fe640 1 -- 172.21.15.161:0/767112889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b20009400 con 0x7f9b300d1cb0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2f7fe640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b300d1cb0 0x7f9b300cdd40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9b200097a0 tx=0x7f9b200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2cff9640 1 -- 172.21.15.161:0/767112889 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b20032910 con 0x7f9b300d1cb0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2cff9640 1 -- 172.21.15.161:0/767112889 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9b20032a70 con 0x7f9b300d1cb0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b2cff9640 1 -- 172.21.15.161:0/767112889 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b20032e20 con 0x7f9b300d1cb0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b3015d700 con 0x7f9b300d1cb0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.822+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b3015fe90 con 0x7f9b300d1cb0 2024-03-23T14:36:23.824 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.823+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b300cc570 con 0x7f9b300d1cb0 2024-03-23T14:36:23.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.823+0000 7f9b2cff9640 1 -- 172.21.15.161:0/767112889 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9b20039040 con 0x7f9b300d1cb0 2024-03-23T14:36:23.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.824+0000 7f9b2cff9640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b1006b810 0x7f9b1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:23.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.824+0000 7f9b2cff9640 1 -- 172.21.15.161:0/767112889 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f9b200ac040 con 0x7f9b300d1cb0 2024-03-23T14:36:23.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.824+0000 7f9b26ffd640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b1006b810 0x7f9b1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:23.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.824+0000 7f9b26ffd640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b1006b810 0x7f9b1006dcd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b300ce7e0 tx=0x7f9b18009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:23.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.828+0000 7f9b2cff9640 1 -- 172.21.15.161:0/767112889 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9b20076980 con 0x7f9b300d1cb0 2024-03-23T14:36:23.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.998+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9b300a2030 con 0x7f9b300d1cb0 2024-03-23T14:36:23.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:23.999+0000 7f9b2cff9640 1 -- 172.21.15.161:0/767112889 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) ==== 104+0+11 (secure 0 0 0) 0x7f9b20037020 con 0x7f9b300d1cb0 2024-03-23T14:36:23.995 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:23.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b1006b810 msgr2=0x7f9b1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b1006b810 0x7f9b1006dcd0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9b300ce7e0 tx=0x7f9b18009040 comp rx=0 tx=0).stop 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b300d1cb0 msgr2=0x7f9b300cdd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b300d1cb0 0x7f9b300cdd40 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f9b200097a0 tx=0x7f9b200046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 shutdown_connections 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9b1006b810 0x7f9b1006dcd0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9b30161b30 0x7f9b300a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9b30161750 0x7f9b30158f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 --2- 172.21.15.161:0/767112889 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9b300d1cb0 0x7f9b300cdd40 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 >> 172.21.15.161:0/767112889 conn(0x7f9b3009d450 msgr2=0x7f9b300b1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:23.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 shutdown_connections 2024-03-23T14:36:23.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.001+0000 7f9b35a6b640 1 -- 172.21.15.161:0/767112889 wait complete. 2024-03-23T14:36:24.008 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:36:24.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.119+0000 7f0e8d981640 1 Processor -- start 2024-03-23T14:36:24.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.119+0000 7f0e8d981640 1 -- start start 2024-03-23T14:36:24.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.119+0000 7f0e8d981640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815dfb0 0x7f0e880d1e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.119+0000 7f0e8d981640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0e8815e390 0x7f0e880d2350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.119+0000 7f0e8d981640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e880cc4f0 con 0x7f0e880d0a40 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.119+0000 7f0e8d981640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f0e880ccab0 con 0x7f0e8815e390 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.119+0000 7f0e8d981640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0e880d1770 con 0x7f0e8815dfb0 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.120+0000 7f0e867fc640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0e8815e390 0x7f0e880d2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.120+0000 7f0e86ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815dfb0 0x7f0e880d1e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.120+0000 7f0e877fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f0e880d0a40 0x7f0e880d0ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:43184/0 (socket says 172.21.15.161:43184) 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.120+0000 7f0e877fe640 1 -- 172.21.15.161:0/4141017937 learned_addr learned my addr 172.21.15.161:0/4141017937 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.120+0000 7f0e867fc640 1 -- 172.21.15.161:0/4141017937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815dfb0 msgr2=0x7f0e880d1e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.120+0000 7f0e867fc640 1 --2- 172.21.15.161:0/4141017937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815dfb0 0x7f0e880d1e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.120+0000 7f0e867fc640 1 -- 172.21.15.161:0/4141017937 >> v1:172.21.15.183:6789/0 conn(0x7f0e880d0a40 legacy=0x7f0e880d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.120+0000 7f0e867fc640 1 -- 172.21.15.161:0/4141017937 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e880d1a50 con 0x7f0e8815e390 2024-03-23T14:36:24.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e867fc640 1 --2- 172.21.15.161:0/4141017937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815e390 0x7f0e880d2350 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0097a0 tx=0x7f0e7c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=641c2fe7dcaafe59 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e85ffb640 1 -- 172.21.15.161:0/4141017937 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c031830 con 0x7f0e8815e390 2024-03-23T14:36:24.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e85ffb640 1 -- 172.21.15.161:0/4141017937 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e7c031990 con 0x7f0e8815e390 2024-03-23T14:36:24.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e85ffb640 1 -- 172.21.15.161:0/4141017937 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7c031d20 con 0x7f0e8815e390 2024-03-23T14:36:24.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e8d981640 1 -- 172.21.15.161:0/4141017937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815e390 msgr2=0x7f0e880d2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e8d981640 1 --2- 172.21.15.161:0/4141017937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815e390 0x7f0e880d2350 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c0097a0 tx=0x7f0e7c02b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e8d981640 1 -- 172.21.15.161:0/4141017937 shutdown_connections 2024-03-23T14:36:24.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e8d981640 1 --2- 172.21.15.161:0/4141017937 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815e390 0x7f0e880d2350 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e8d981640 1 --2- 172.21.15.161:0/4141017937 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815dfb0 0x7f0e880d1e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e8d981640 1 -- 172.21.15.161:0/4141017937 >> 172.21.15.161:0/4141017937 conn(0x7f0e8809aa40 msgr2=0x7f0e8809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e8d981640 1 -- 172.21.15.161:0/4141017937 shutdown_connections 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.121+0000 7f0e8d981640 1 -- 172.21.15.161:0/4141017937 wait complete. 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.123+0000 7f0e8d981640 1 Processor -- start 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.123+0000 7f0e8d981640 1 -- start start 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.123+0000 7f0e8d981640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e880d0a40 0x7f0e880cb260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.123+0000 7f0e8d981640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815dfb0 0x7f0e880c82d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.123+0000 7f0e8d981640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815e390 0x7f0e88158430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.123+0000 7f0e8d981640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e880c8810 con 0x7f0e880d0a40 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.123+0000 7f0e8d981640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e88158970 con 0x7f0e8815dfb0 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.123+0000 7f0e8d981640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0e880cbf20 con 0x7f0e8815e390 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e880d0a40 0x7f0e880cb260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e880d0a40 0x7f0e880cb260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51604/0 (socket says 172.21.15.161:51604) 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e867fc640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815dfb0 0x7f0e880c82d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 -- 172.21.15.161:0/3129859130 learned_addr learned my addr 172.21.15.161:0/3129859130 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e877fe640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815e390 0x7f0e88158430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 -- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815e390 msgr2=0x7f0e88158430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815e390 0x7f0e88158430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 -- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815dfb0 msgr2=0x7f0e880c82d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815dfb0 0x7f0e880c82d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 -- 172.21.15.161:0/3129859130 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e7c009400 con 0x7f0e880d0a40 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e867fc640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815dfb0 0x7f0e880c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e86ffd640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e880d0a40 0x7f0e880cb260 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800e9e0 tx=0x7f0e7800eea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e6ffff640 1 -- 172.21.15.161:0/3129859130 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78009670 con 0x7f0e880d0a40 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e6ffff640 1 -- 172.21.15.161:0/3129859130 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e7800c9d0 con 0x7f0e880d0a40 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e6ffff640 1 -- 172.21.15.161:0/3129859130 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e7800cd80 con 0x7f0e880d0a40 2024-03-23T14:36:24.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e880c5e70 con 0x7f0e880d0a40 2024-03-23T14:36:24.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.124+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e880cb960 con 0x7f0e880d0a40 2024-03-23T14:36:24.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.125+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e8804f4e0 con 0x7f0e880d0a40 2024-03-23T14:36:24.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.126+0000 7f0e6ffff640 1 -- 172.21.15.161:0/3129859130 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0e78013470 con 0x7f0e880d0a40 2024-03-23T14:36:24.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.126+0000 7f0e6ffff640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e7406b840 0x7f0e7406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.126+0000 7f0e6ffff640 1 -- 172.21.15.161:0/3129859130 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f0e7808a550 con 0x7f0e880d0a40 2024-03-23T14:36:24.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.126+0000 7f0e867fc640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e7406b840 0x7f0e7406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.127+0000 7f0e867fc640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e7406b840 0x7f0e7406dd00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c036000 tx=0x7f0e7c03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.131+0000 7f0e6ffff640 1 -- 172.21.15.161:0/3129859130 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0e78054e10 con 0x7f0e880d0a40 2024-03-23T14:36:24.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.303+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0e88161740 con 0x7f0e880d0a40 2024-03-23T14:36:24.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.303+0000 7f0e6ffff640 1 -- 172.21.15.161:0/3129859130 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) ==== 100+0+11 (secure 0 0 0) 0x7f0e780584d0 con 0x7f0e880d0a40 2024-03-23T14:36:24.299 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:36:24.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e7406b840 msgr2=0x7f0e7406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e7406b840 0x7f0e7406dd00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f0e7c036000 tx=0x7f0e7c03c040 comp rx=0 tx=0).stop 2024-03-23T14:36:24.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e880d0a40 msgr2=0x7f0e880cb260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e880d0a40 0x7f0e880cb260 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f0e7800e9e0 tx=0x7f0e7800eea0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 shutdown_connections 2024-03-23T14:36:24.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e7406b840 0x7f0e7406dd00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e8815e390 0x7f0e88158430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e8815dfb0 0x7f0e880c82d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 --2- 172.21.15.161:0/3129859130 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e880d0a40 0x7f0e880cb260 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 >> 172.21.15.161:0/3129859130 conn(0x7f0e8809aa40 msgr2=0x7f0e8806ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:24.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 shutdown_connections 2024-03-23T14:36:24.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.305+0000 7f0e8d981640 1 -- 172.21.15.161:0/3129859130 wait complete. 2024-03-23T14:36:24.312 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-23T14:36:24.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.420+0000 7f92bb5e2640 1 Processor -- start 2024-03-23T14:36:24.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.421+0000 7f92bb5e2640 1 -- start start 2024-03-23T14:36:24.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.421+0000 7f92bb5e2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f92b40b43f0 0x7f92b415b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.421+0000 7f92bb5e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b415b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.421+0000 7f92bb5e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92b415c1f0 con 0x7f92b40b4010 2024-03-23T14:36:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.421+0000 7f92bb5e2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f92b406ef20 con 0x7f92b40b43f0 2024-03-23T14:36:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.421+0000 7f92bb5e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f92b415f730 con 0x7f92b40d05f0 2024-03-23T14:36:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92b8b7d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f92b40b43f0 0x7f92b415b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92b9b7f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b415b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92b937e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f92b40b4010 0x7f92b40cfee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:43186/0 (socket says 172.21.15.161:43186) 2024-03-23T14:36:24.420 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92b937e640 1 -- 172.21.15.161:0/1183631323 learned_addr learned my addr 172.21.15.161:0/1183631323 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92abfff640 1 -- 172.21.15.161:0/1183631323 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3751602465 0 0) 0x7f92b415c1f0 con 0x7f92b40b4010 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92abfff640 1 -- 172.21.15.161:0/1183631323 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92a0003520 con 0x7f92b40b4010 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92b9b7f640 1 -- 172.21.15.161:0/1183631323 >> v2:172.21.15.183:3300/0 conn(0x7f92b40b43f0 msgr2=0x7f92b415b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92b9b7f640 1 --2- 172.21.15.161:0/1183631323 >> v2:172.21.15.183:3300/0 conn(0x7f92b40b43f0 0x7f92b415b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92b9b7f640 1 -- 172.21.15.161:0/1183631323 >> v1:172.21.15.183:6789/0 conn(0x7f92b40b4010 legacy=0x7f92b40cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.422+0000 7f92b9b7f640 1 -- 172.21.15.161:0/1183631323 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b415e970 con 0x7f92b40d05f0 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92b9b7f640 1 --2- 172.21.15.161:0/1183631323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b415b9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f92b00097d0 tx=0x7f92b002b810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f269fb41ce3931df server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92abfff640 1 -- 172.21.15.161:0/1183631323 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0031770 con 0x7f92b40d05f0 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92abfff640 1 -- 172.21.15.161:0/1183631323 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92b00318d0 con 0x7f92b40d05f0 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92abfff640 1 -- 172.21.15.161:0/1183631323 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92b0031c60 con 0x7f92b40d05f0 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92bb5e2640 1 -- 172.21.15.161:0/1183631323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 msgr2=0x7f92b415b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/1183631323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b415b9c0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f92b00097d0 tx=0x7f92b002b810 comp rx=0 tx=0).stop 2024-03-23T14:36:24.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92bb5e2640 1 -- 172.21.15.161:0/1183631323 shutdown_connections 2024-03-23T14:36:24.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/1183631323 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b415b9c0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/1183631323 >> v2:172.21.15.183:3300/0 conn(0x7f92b40b43f0 0x7f92b415b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.423+0000 7f92bb5e2640 1 -- 172.21.15.161:0/1183631323 >> 172.21.15.161:0/1183631323 conn(0x7f92b406b4a0 msgr2=0x7f92b406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:24.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.424+0000 7f92bb5e2640 1 -- 172.21.15.161:0/1183631323 shutdown_connections 2024-03-23T14:36:24.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.424+0000 7f92bb5e2640 1 -- 172.21.15.161:0/1183631323 wait complete. 2024-03-23T14:36:24.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92bb5e2640 1 Processor -- start 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92bb5e2640 1 -- start start 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92bb5e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b40b4010 0x7f92b413c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92bb5e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b40b43f0 0x7f92b414ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92bb5e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b41504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92bb5e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92b413caf0 con 0x7f92b40b4010 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92bb5e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f92b41510f0 con 0x7f92b40b43f0 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92bb5e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f92b4150e10 con 0x7f92b40d05f0 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92b937e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b40b4010 0x7f92b413c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92b937e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b40b4010 0x7f92b413c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51614/0 (socket says 172.21.15.161:51614) 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.425+0000 7f92b937e640 1 -- 172.21.15.161:0/805522296 learned_addr learned my addr 172.21.15.161:0/805522296 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92b9b7f640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b41504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92b8b7d640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b40b43f0 0x7f92b414ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92b937e640 1 -- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 msgr2=0x7f92b41504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92b937e640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b41504d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92b937e640 1 -- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b40b43f0 msgr2=0x7f92b414ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92b937e640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b40b43f0 0x7f92b414ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92b937e640 1 -- 172.21.15.161:0/805522296 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b0009400 con 0x7f92b40b4010 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92b937e640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b40b4010 0x7f92b413c5b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00cb40 tx=0x7f92ac00a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92a9ffb640 1 -- 172.21.15.161:0/805522296 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac00aca0 con 0x7f92b40b4010 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92a9ffb640 1 -- 172.21.15.161:0/805522296 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92ac00ae00 con 0x7f92b40b4010 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92a9ffb640 1 -- 172.21.15.161:0/805522296 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92ac012620 con 0x7f92b40b4010 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b40cda00 con 0x7f92b40b4010 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.426+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b40cdfc0 con 0x7f92b40b4010 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.427+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92b40cd860 con 0x7f92b40b4010 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.428+0000 7f92a9ffb640 1 -- 172.21.15.161:0/805522296 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f92ac007070 con 0x7f92b40b4010 2024-03-23T14:36:24.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.428+0000 7f92a9ffb640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f929806b810 0x7f929806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.428+0000 7f92a9ffb640 1 -- 172.21.15.161:0/805522296 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f92ac08a350 con 0x7f92b40b4010 2024-03-23T14:36:24.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.428+0000 7f92b8b7d640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f929806b810 0x7f929806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.429+0000 7f92b8b7d640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f929806b810 0x7f929806dcd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92b414ee30 tx=0x7f92a4009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.432+0000 7f92a9ffb640 1 -- 172.21.15.161:0/805522296 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f92b40cd860 con 0x7f92b40b4010 2024-03-23T14:36:24.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.602+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f92b40cd860 con 0x7f92b40b4010 2024-03-23T14:36:24.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.602+0000 7f92a9ffb640 1 -- 172.21.15.161:0/805522296 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v36) ==== 97+0+11 (secure 0 0 0) 0x7f92b40cd860 con 0x7f92b40b4010 2024-03-23T14:36:24.598 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:24.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f929806b810 msgr2=0x7f929806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f929806b810 0x7f929806dcd0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f92b414ee30 tx=0x7f92a4009190 comp rx=0 tx=0).stop 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b40b4010 msgr2=0x7f92b413c5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b40b4010 0x7f92b413c5b0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f92ac00cb40 tx=0x7f92ac00a060 comp rx=0 tx=0).stop 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 shutdown_connections 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f929806b810 0x7f929806dcd0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f92b40d05f0 0x7f92b41504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92b40b43f0 0x7f92b414ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 --2- 172.21.15.161:0/805522296 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92b40b4010 0x7f92b413c5b0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 >> 172.21.15.161:0/805522296 conn(0x7f92b406b4a0 msgr2=0x7f92b40cfee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:24.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 shutdown_connections 2024-03-23T14:36:24.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.604+0000 7f92bb5e2640 1 -- 172.21.15.161:0/805522296 wait complete. 2024-03-23T14:36:24.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:36:24.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.718+0000 7fc0e68d4640 1 Processor -- start 2024-03-23T14:36:24.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.718+0000 7fc0e68d4640 1 -- start start 2024-03-23T14:36:24.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.719+0000 7fc0e68d4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc0e00724e0 0x7fc0e006e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.719+0000 7fc0e68d4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e015be70 con 0x7fc0e0072100 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.719+0000 7fc0e68d4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0e015c150 con 0x7fc0e006ead0 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.719+0000 7fc0e68d4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc0e006ef80 con 0x7fc0e00724e0 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.719+0000 7fc0df7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc0e00724e0 0x7fc0e006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.719+0000 7fc0dffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc0e0072100 0x7fc0e015b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50036/0 (socket says 172.21.15.161:50036) 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.719+0000 7fc0dffff640 1 -- 172.21.15.161:0/2766210586 learned_addr learned my addr 172.21.15.161:0/2766210586 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.720+0000 7fc0df7fe640 1 -- 172.21.15.161:0/2766210586 >> v1:172.21.15.183:6789/0 conn(0x7fc0e006ead0 legacy=0x7fc0e00b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.720+0000 7fc0df7fe640 1 -- 172.21.15.161:0/2766210586 >> v1:172.21.15.161:6789/0 conn(0x7fc0e0072100 legacy=0x7fc0e015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.720+0000 7fc0df7fe640 1 -- 172.21.15.161:0/2766210586 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e00b2c70 con 0x7fc0e00724e0 2024-03-23T14:36:24.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.720+0000 7fc0deffd640 1 -- 172.21.15.161:0/2766210586 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4251124830 0 0) 0x7fc0e015be70 con 0x7fc0e0072100 2024-03-23T14:36:24.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.720+0000 7fc0df7fe640 1 --2- 172.21.15.161:0/2766210586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e00724e0 0x7fc0e006e590 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40097a0 tx=0x7fc0d402b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5cc58b576769738d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0deffd640 1 -- 172.21.15.161:0/2766210586 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4030610 con 0x7fc0e00724e0 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0deffd640 1 -- 172.21.15.161:0/2766210586 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d4030770 con 0x7fc0e00724e0 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0deffd640 1 -- 172.21.15.161:0/2766210586 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4030b00 con 0x7fc0e00724e0 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0e68d4640 1 -- 172.21.15.161:0/2766210586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e00724e0 msgr2=0x7fc0e006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0e68d4640 1 --2- 172.21.15.161:0/2766210586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e00724e0 0x7fc0e006e590 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fc0d40097a0 tx=0x7fc0d402b960 comp rx=0 tx=0).stop 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0e68d4640 1 -- 172.21.15.161:0/2766210586 shutdown_connections 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0e68d4640 1 --2- 172.21.15.161:0/2766210586 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e00724e0 0x7fc0e006e590 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0e68d4640 1 -- 172.21.15.161:0/2766210586 >> 172.21.15.161:0/2766210586 conn(0x7fc0e00b17f0 msgr2=0x7fc0e00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0e68d4640 1 -- 172.21.15.161:0/2766210586 shutdown_connections 2024-03-23T14:36:24.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.721+0000 7fc0e68d4640 1 -- 172.21.15.161:0/2766210586 wait complete. 2024-03-23T14:36:24.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.722+0000 7fc0e68d4640 1 Processor -- start 2024-03-23T14:36:24.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0e68d4640 1 -- start start 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0e68d4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e006ead0 0x7fc0e014e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0e68d4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0e0072100 0x7fc0e014f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0e68d4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0e00724e0 0x7fc0e014cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0e68d4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e014d510 con 0x7fc0e006ead0 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0e68d4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e014d7f0 con 0x7fc0e00724e0 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0e68d4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc0e014dad0 con 0x7fc0e0072100 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e006ead0 0x7fc0e014e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0dffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e006ead0 0x7fc0e014e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51644/0 (socket says 172.21.15.161:51644) 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0dffff640 1 -- 172.21.15.161:0/3827209133 learned_addr learned my addr 172.21.15.161:0/3827209133 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0e4e71640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0e00724e0 0x7fc0e014cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.723+0000 7fc0df7fe640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0e0072100 0x7fc0e014f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dffff640 1 -- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0e0072100 msgr2=0x7fc0e014f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dffff640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0e0072100 0x7fc0e014f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dffff640 1 -- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0e00724e0 msgr2=0x7fc0e014cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dffff640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0e00724e0 0x7fc0e014cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dffff640 1 -- 172.21.15.161:0/3827209133 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0d4009400 con 0x7fc0e006ead0 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dffff640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e006ead0 0x7fc0e014e5b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000e730 tx=0x7fc0d000ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dcff9640 1 -- 172.21.15.161:0/3827209133 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d000ee40 con 0x7fc0e006ead0 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dcff9640 1 -- 172.21.15.161:0/3827209133 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0d00072e0 con 0x7fc0e006ead0 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e014ddb0 con 0x7fc0e006ead0 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0dcff9640 1 -- 172.21.15.161:0/3827209133 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d0007690 con 0x7fc0e006ead0 2024-03-23T14:36:24.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.724+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e014c390 con 0x7fc0e006ead0 2024-03-23T14:36:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.725+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e00735e0 con 0x7fc0e006ead0 2024-03-23T14:36:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.725+0000 7fc0dcff9640 1 -- 172.21.15.161:0/3827209133 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc0d000c7f0 con 0x7fc0e006ead0 2024-03-23T14:36:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.726+0000 7fc0dcff9640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0cc06b810 0x7fc0cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.726+0000 7fc0dcff9640 1 -- 172.21.15.161:0/3827209133 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fc0d008c570 con 0x7fc0e006ead0 2024-03-23T14:36:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.726+0000 7fc0df7fe640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0cc06b810 0x7fc0cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:24.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.726+0000 7fc0df7fe640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0cc06b810 0x7fc0cc06dcd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4036000 tx=0x7fc0d403b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:24.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.730+0000 7fc0dcff9640 1 -- 172.21.15.161:0/3827209133 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc0d0056eb0 con 0x7fc0e006ead0 2024-03-23T14:36:24.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.912+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fc0e004f550 con 0x7fc0e006ead0 2024-03-23T14:36:24.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.912+0000 7fc0dcff9640 1 -- 172.21.15.161:0/3827209133 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) ==== 80+0+0 (secure 0 0 0) 0x7fc0d005a570 con 0x7fc0e006ead0 2024-03-23T14:36:24.908 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0cc06b810 msgr2=0x7fc0cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0cc06b810 0x7fc0cc06dcd0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4036000 tx=0x7fc0d403b040 comp rx=0 tx=0).stop 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e006ead0 msgr2=0x7fc0e014e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e006ead0 0x7fc0e014e5b0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fc0d000e730 tx=0x7fc0d000ebf0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 shutdown_connections 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc0cc06b810 0x7fc0cc06dcd0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0e00724e0 0x7fc0e014cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0e0072100 0x7fc0e014f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 --2- 172.21.15.161:0/3827209133 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0e006ead0 0x7fc0e014e5b0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 >> 172.21.15.161:0/3827209133 conn(0x7fc0e00b17f0 msgr2=0x7fc0e00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 shutdown_connections 2024-03-23T14:36:24.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:24.914+0000 7fc0e68d4640 1 -- 172.21.15.161:0/3827209133 wait complete. 2024-03-23T14:36:24.921 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-23T14:36:24.922 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:24.922 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:24.949 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[51586]: fuse finished with error 0 and tester_r 0 2024-03-23T14:36:24.951 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:36:25.009 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:36:25.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:25.010 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T14:36:25.027 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:36:25.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.173+0000 7fb903e55640 1 Processor -- start 2024-03-23T14:36:25.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.174+0000 7fb903e55640 1 -- start start 2024-03-23T14:36:25.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.174+0000 7fb903e55640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb8fc15e400 0x7fb8fc0cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:25.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.174+0000 7fb903e55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0d2b40 0x7fb8fc0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:25.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.174+0000 7fb903e55640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8fc0d0460 con 0x7fb8fc0cf9d0 2024-03-23T14:36:25.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.174+0000 7fb903e55640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fb8fc0d1050 con 0x7fb8fc15e400 2024-03-23T14:36:25.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.174+0000 7fb903e55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb8fc0d06b0 con 0x7fb8fc0d2b40 2024-03-23T14:36:25.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb9023f2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0d2b40 0x7fb8fc0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:25.171 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb9023f2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0d2b40 0x7fb8fc0b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:59404/0 (socket says 172.21.15.161:59404) 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb9013f0640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb8fc0cf9d0 0x7fb8fc0d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:43192/0 (socket says 172.21.15.161:43192) 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb9023f2640 1 -- 172.21.15.161:0/1966692073 learned_addr learned my addr 172.21.15.161:0/1966692073 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb901bf1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb8fc15e400 0x7fb8fc0cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb9023f2640 1 -- 172.21.15.161:0/1966692073 >> v2:172.21.15.183:3300/0 conn(0x7fb8fc15e400 msgr2=0x7fb8fc0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb9023f2640 1 --2- 172.21.15.161:0/1966692073 >> v2:172.21.15.183:3300/0 conn(0x7fb8fc15e400 0x7fb8fc0cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb9023f2640 1 -- 172.21.15.161:0/1966692073 >> v1:172.21.15.183:6789/0 conn(0x7fb8fc0cf9d0 legacy=0x7fb8fc0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb900bef640 1 -- 172.21.15.161:0/1966692073 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3856547881 0 0) 0x7fb8fc0d0460 con 0x7fb8fc0cf9d0 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb9023f2640 1 -- 172.21.15.161:0/1966692073 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8fc0d0990 con 0x7fb8fc0d2b40 2024-03-23T14:36:25.172 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.175+0000 7fb901bf1640 1 --2- 172.21.15.161:0/1966692073 >> v2:172.21.15.183:3300/0 conn(0x7fb8fc15e400 0x7fb8fc0cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.176+0000 7fb9023f2640 1 --2- 172.21.15.161:0/1966692073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0d2b40 0x7fb8fc0b0a80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009870 tx=0x7fb8f402b980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10a5ebfbb7b83bed server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.176+0000 7fb900bef640 1 -- 172.21.15.161:0/1966692073 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4030840 con 0x7fb8fc0d2b40 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.176+0000 7fb900bef640 1 -- 172.21.15.161:0/1966692073 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8f40309a0 con 0x7fb8fc0d2b40 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.176+0000 7fb900bef640 1 -- 172.21.15.161:0/1966692073 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4030d30 con 0x7fb8fc0d2b40 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.177+0000 7fb903e55640 1 -- 172.21.15.161:0/1966692073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0d2b40 msgr2=0x7fb8fc0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.177+0000 7fb903e55640 1 --2- 172.21.15.161:0/1966692073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0d2b40 0x7fb8fc0b0a80 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009870 tx=0x7fb8f402b980 comp rx=0 tx=0).stop 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.177+0000 7fb903e55640 1 -- 172.21.15.161:0/1966692073 shutdown_connections 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.177+0000 7fb903e55640 1 --2- 172.21.15.161:0/1966692073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0d2b40 0x7fb8fc0b0a80 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.177+0000 7fb903e55640 1 --2- 172.21.15.161:0/1966692073 >> v2:172.21.15.183:3300/0 conn(0x7fb8fc15e400 0x7fb8fc0cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.177+0000 7fb903e55640 1 -- 172.21.15.161:0/1966692073 >> 172.21.15.161:0/1966692073 conn(0x7fb8fc09d190 msgr2=0x7fb8fc0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.177+0000 7fb903e55640 1 -- 172.21.15.161:0/1966692073 shutdown_connections 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.177+0000 7fb903e55640 1 -- 172.21.15.161:0/1966692073 wait complete. 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.178+0000 7fb903e55640 1 Processor -- start 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.178+0000 7fb903e55640 1 -- start start 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb903e55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0cf9d0 0x7fb8fc0c8910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb903e55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8fc0d2b40 0x7fb8fc155a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb903e55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc15e400 0x7fb8fc15abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb903e55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb8fc0c94f0 con 0x7fb8fc15e400 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb903e55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb8fc15fe80 con 0x7fb8fc0d2b40 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb903e55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb8fc160720 con 0x7fb8fc0cf9d0 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc15e400 0x7fb8fc15abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc15e400 0x7fb8fc15abd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51648/0 (socket says 172.21.15.161:51648) 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 -- 172.21.15.161:0/2027119074 learned_addr learned my addr 172.21.15.161:0/2027119074 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:25.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9013f0640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8fc0d2b40 0x7fb8fc155a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb901bf1640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0cf9d0 0x7fb8fc0c8910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 -- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0cf9d0 msgr2=0x7fb8fc0c8910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0cf9d0 0x7fb8fc0c8910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 -- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8fc0d2b40 msgr2=0x7fb8fc155a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8fc0d2b40 0x7fb8fc155a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 -- 172.21.15.161:0/2027119074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8ec009400 con 0x7fb8fc15e400 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb901bf1640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0cf9d0 0x7fb8fc0c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb9023f2640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc15e400 0x7fb8fc15abd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009870 tx=0x7fb8f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb8fa7fc640 1 -- 172.21.15.161:0/2027119074 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4039c90 con 0x7fb8fc15e400 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb8fa7fc640 1 -- 172.21.15.161:0/2027119074 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb8f4039e10 con 0x7fb8fc15e400 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb8fa7fc640 1 -- 172.21.15.161:0/2027119074 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8f4043640 con 0x7fb8fc15e400 2024-03-23T14:36:25.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.179+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8f40094d0 con 0x7fb8fc15e400 2024-03-23T14:36:25.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.180+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8fc0c9210 con 0x7fb8fc15e400 2024-03-23T14:36:25.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.180+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8fc0a1e00 con 0x7fb8fc15e400 2024-03-23T14:36:25.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.181+0000 7fb8fa7fc640 1 -- 172.21.15.161:0/2027119074 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb8f40437a0 con 0x7fb8fc15e400 2024-03-23T14:36:25.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.181+0000 7fb8fa7fc640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8e806b840 0x7fb8e806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:25.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.181+0000 7fb8fa7fc640 1 -- 172.21.15.161:0/2027119074 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fb8f403e070 con 0x7fb8fc15e400 2024-03-23T14:36:25.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.181+0000 7fb901bf1640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8e806b840 0x7fb8e806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:25.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.182+0000 7fb901bf1640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8e806b840 0x7fb8e806dd00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec009fd0 tx=0x7fb8ec009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:25.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.185+0000 7fb8fa7fc640 1 -- 172.21.15.161:0/2027119074 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb8f4077b40 con 0x7fb8fc15e400 2024-03-23T14:36:25.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.394+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb8fc160860 con 0x7fb8fc15e400 2024-03-23T14:36:25.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.395+0000 7fb8fa7fc640 1 -- 172.21.15.161:0/2027119074 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+82 (secure 0 0 0) 0x7fb8f407b200 con 0x7fb8fc15e400 2024-03-23T14:36:25.391 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8e806b840 msgr2=0x7fb8e806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8e806b840 0x7fb8e806dd00 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb8ec009fd0 tx=0x7fb8ec009190 comp rx=0 tx=0).stop 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc15e400 msgr2=0x7fb8fc15abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc15e400 0x7fb8fc15abd0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fb8f4009870 tx=0x7fb8f4004730 comp rx=0 tx=0).stop 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 shutdown_connections 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8e806b840 0x7fb8e806dd00 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb8fc15e400 0x7fb8fc15abd0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb8fc0d2b40 0x7fb8fc155a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 --2- 172.21.15.161:0/2027119074 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb8fc0cf9d0 0x7fb8fc0c8910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 >> 172.21.15.161:0/2027119074 conn(0x7fb8fc09d190 msgr2=0x7fb8fc162620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:25.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 shutdown_connections 2024-03-23T14:36:25.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25.397+0000 7fb903e55640 1 -- 172.21.15.161:0/2027119074 wait complete. 2024-03-23T14:36:25.403 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:36:25.403 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:36:25.405 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-23T14:36:25.405 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T14:36:25.405 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:25.405 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:36:25.405 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:36:25.405 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:36:25.405 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:25.406 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:25.406 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-23T14:36:25.421 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:36:25.421 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:25.422 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42684sec preferred_lft 42684sec 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c4a9:64ff:fe59:877a/64 scope link 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c48:50ff:fe01:9f4e/64 scope link 2024-03-23T14:36:25.423 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:25.424 INFO:teuthology.orchestra.run.smithi161.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:36:25.424 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 62:02:d5:17:e7:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:25.424 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::6002:d5ff:fe17:e70d/64 scope link 2024-03-23T14:36:25.424 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:25.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:25.424 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:36:25.490 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-23T14:36:25.490 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:25.490 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-23T14:36:25.549 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-23T14:36:25.549 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:36:25.549 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:36:25.549 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:36:25.549 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:36:25.550 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:36:25.550 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:36:25.550 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:36:25.550 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:36:25.550 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:36:25.550 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:36:25.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:36:25.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:25 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:36:25.685 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:25.685 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:36:25.701 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T14:36:25.701 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T14:36:25.701 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:25.701 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:25.702 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-23T14:36:25.770 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T14:36:25.770 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:36:25.770 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:25.828 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:25.829 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:36:25.829 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:25.883 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:36:25.951 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:36:26.007 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:36:26.008 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:36:26.009 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:36:26.010 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:36:26.010 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:36:26.010 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:36:26.010 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:26.010 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:26.010 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:26.010 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:26.010 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:26.010 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:36:26.062 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T14:36:26.062 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T14:36:26.105 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:36:26.132 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:36:26.187 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:36:26.188 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:36:26.188 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:36:26.188 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:36:26.188 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:36:26.188 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:36:26.188 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:36:26.189 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:36:26.190 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:36:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:26.191 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:26.191 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:26.191 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:36:26.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:2024-03-23T14:36:26.195+0000 7f6bfee37480 -1 init, newargv = 0x55588ce768f0 newargc=15 2024-03-23T14:36:26.192 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[52821]: starting ceph client 2024-03-23T14:36:26.195 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[52821]: starting fuse 2024-03-23T14:36:26.206 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:36:26.207 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-23T14:36:26.207 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:36:26.208 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:26.209 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.0")) 2024-03-23T14:36:26.209 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:36:26.326 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.0.52821.asok 2024-03-23T14:36:26.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:36:26.336 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.52821.asok 2024-03-23T14:36:26.336 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:26.336 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.52821.asok status 2024-03-23T14:36:26.472 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:36:26.472 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "0", 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "52821", 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5580, 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-23T14:36:26.473 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 5580 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 2328776528 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 2328776528 2024-03-23T14:36:26.474 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.5580 192.168.0.1:0/2328776528", 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/2328776528", 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 16, 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 36, 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-23T14:36:26.475 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:36:26.480 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:26.481 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-23T14:36:26.481 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:26.496 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-23T14:36:26.496 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:26.496 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:26.496 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:26.564 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:26.564 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-23T14:36:26.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:36:26.630 INFO:teuthology.orchestra.run.smithi161.stderr:setfattr: layoutfile: Permission denied 2024-03-23T14:36:26.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:26.631 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-23T14:36:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:36:26.687 INFO:teuthology.orchestra.run.smithi161.stderr:setfattr: layoutdir: Permission denied 2024-03-23T14:36:26.687 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:26.687 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:26.743 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:26.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.888+0000 7f5e3ead0640 1 Processor -- start 2024-03-23T14:36:26.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.888+0000 7f5e3ead0640 1 -- start start 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3ead0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5e3806eb30 0x7f5e380b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3ead0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5e380b3610 0x7f5e380b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3ead0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e380b3fa0 con 0x7f5e380b4260 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3ead0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5e3815ab90 con 0x7f5e3806eb30 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3ead0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f5e380713a0 con 0x7f5e380b3610 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3c86c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5e3806eb30 0x7f5e380b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3c86c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5e3806eb30 0x7f5e380b4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51654/0 (socket says 172.21.15.161:51654) 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3c86c640 1 -- 172.21.15.161:0/1147649691 learned_addr learned my addr 172.21.15.161:0/1147649691 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3d06d640 1 --2- 172.21.15.161:0/1147649691 >> v2:172.21.15.183:3300/0 conn(0x7f5e380b3610 0x7f5e380b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3c86c640 1 -- 172.21.15.161:0/1147649691 >> v2:172.21.15.183:3300/0 conn(0x7f5e380b3610 msgr2=0x7f5e380b3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:26.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3c86c640 1 --2- 172.21.15.161:0/1147649691 >> v2:172.21.15.183:3300/0 conn(0x7f5e380b3610 0x7f5e380b3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3c86c640 1 -- 172.21.15.161:0/1147649691 >> v1:172.21.15.183:6789/0 conn(0x7f5e380b4260 legacy=0x7f5e3815b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.889+0000 7f5e3c86c640 1 -- 172.21.15.161:0/1147649691 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e38070b90 con 0x7f5e3806eb30 2024-03-23T14:36:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e3c86c640 1 --2- 172.21.15.161:0/1147649691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e3806eb30 0x7f5e380b4870 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5e3806c9b0 tx=0x7f5e3002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=922efce14cfedf2d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e377fe640 1 -- 172.21.15.161:0/1147649691 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3002ca80 con 0x7f5e3806eb30 2024-03-23T14:36:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e377fe640 1 -- 172.21.15.161:0/1147649691 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e3002cbe0 con 0x7f5e3806eb30 2024-03-23T14:36:26.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e377fe640 1 -- 172.21.15.161:0/1147649691 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e300343f0 con 0x7f5e3806eb30 2024-03-23T14:36:26.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e3ead0640 1 -- 172.21.15.161:0/1147649691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e3806eb30 msgr2=0x7f5e380b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:26.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/1147649691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e3806eb30 0x7f5e380b4870 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5e3806c9b0 tx=0x7f5e3002bee0 comp rx=0 tx=0).stop 2024-03-23T14:36:26.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e3ead0640 1 -- 172.21.15.161:0/1147649691 shutdown_connections 2024-03-23T14:36:26.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/1147649691 >> v2:172.21.15.183:3300/0 conn(0x7f5e380b3610 0x7f5e380b3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/1147649691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e3806eb30 0x7f5e380b4870 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.890+0000 7f5e3ead0640 1 -- 172.21.15.161:0/1147649691 >> 172.21.15.161:0/1147649691 conn(0x7f5e380b17f0 msgr2=0x7f5e380b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.891+0000 7f5e3ead0640 1 -- 172.21.15.161:0/1147649691 shutdown_connections 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.891+0000 7f5e3ead0640 1 -- 172.21.15.161:0/1147649691 wait complete. 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.892+0000 7f5e3ead0640 1 Processor -- start 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.892+0000 7f5e3ead0640 1 -- start start 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3ead0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e3806eb30 0x7f5e38149d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3ead0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e380b3610 0x7f5e381f1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3ead0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e380b4260 0x7f5e38149530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3ead0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5e381f1e90 con 0x7f5e380b3610 2024-03-23T14:36:26.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3ead0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3814a8e0 con 0x7f5e380b4260 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3ead0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3814abc0 con 0x7f5e3806eb30 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e380b3610 0x7f5e381f1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e380b3610 0x7f5e381f1950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51662/0 (socket says 172.21.15.161:51662) 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 -- 172.21.15.161:0/2150604784 learned_addr learned my addr 172.21.15.161:0/2150604784 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3c86c640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e3806eb30 0x7f5e38149d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3d06d640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e380b4260 0x7f5e38149530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 -- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e3806eb30 msgr2=0x7f5e38149d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e3806eb30 0x7f5e38149d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 -- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e380b4260 msgr2=0x7f5e38149530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e380b4260 0x7f5e38149530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 -- 172.21.15.161:0/2150604784 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e30009b30 con 0x7f5e380b3610 2024-03-23T14:36:26.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e37fff640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e380b3610 0x7f5e381f1950 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000e710 tx=0x7f5e2000ebd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e357fa640 1 -- 172.21.15.161:0/2150604784 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e200052d0 con 0x7f5e380b3610 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e357fa640 1 -- 172.21.15.161:0/2150604784 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e20005430 con 0x7f5e380b3610 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e3814f7b0 con 0x7f5e380b3610 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e357fa640 1 -- 172.21.15.161:0/2150604784 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e20001ab0 con 0x7f5e380b3610 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.893+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e38149a70 con 0x7f5e380b3610 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.894+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e3814aa20 con 0x7f5e380b3610 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.895+0000 7f5e357fa640 1 -- 172.21.15.161:0/2150604784 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5e20001c10 con 0x7f5e380b3610 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.895+0000 7f5e357fa640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e1806b810 0x7f5e1806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:26.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.895+0000 7f5e357fa640 1 -- 172.21.15.161:0/2150604784 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f5e2008c1d0 con 0x7f5e380b3610 2024-03-23T14:36:26.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.895+0000 7f5e3c86c640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e1806b810 0x7f5e1806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:26.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.896+0000 7f5e3c86c640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e1806b810 0x7f5e1806dcd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5e30035000 tx=0x7f5e30030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:26.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:26.899+0000 7f5e357fa640 1 -- 172.21.15.161:0/2150604784 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5e3814aa20 con 0x7f5e380b3610 2024-03-23T14:36:27.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.074+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5e380b3fa0 con 0x7f5e380b3610 2024-03-23T14:36:27.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.075+0000 7f5e357fa640 1 -- 172.21.15.161:0/2150604784 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7f5e380b3fa0 con 0x7f5e380b3610 2024-03-23T14:36:27.071 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:27.072 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:27.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e1806b810 msgr2=0x7f5e1806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e1806b810 0x7f5e1806dcd0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5e30035000 tx=0x7f5e30030000 comp rx=0 tx=0).stop 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e380b3610 msgr2=0x7f5e381f1950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e380b3610 0x7f5e381f1950 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f5e2000e710 tx=0x7f5e2000ebd0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 shutdown_connections 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e1806b810 0x7f5e1806dcd0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e380b4260 0x7f5e38149530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e380b3610 0x7f5e381f1950 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 --2- 172.21.15.161:0/2150604784 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e3806eb30 0x7f5e38149d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.077+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 >> 172.21.15.161:0/2150604784 conn(0x7f5e380b17f0 msgr2=0x7f5e3815b560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.078+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 shutdown_connections 2024-03-23T14:36:27.074 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.078+0000 7f5e3ead0640 1 -- 172.21.15.161:0/2150604784 wait complete. 2024-03-23T14:36:27.085 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:04.688180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:05.241774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-23T14:36:18.738957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:27.085 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:27.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.193+0000 7f6360e2e640 1 Processor -- start 2024-03-23T14:36:27.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.193+0000 7f6360e2e640 1 -- start start 2024-03-23T14:36:27.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.194+0000 7f6360e2e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d1270 0x7f635c0d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.194+0000 7f6360e2e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f635c0d29d0 0x7f635c0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.194+0000 7f6360e2e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f635c161c00 con 0x7f635c161820 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.194+0000 7f6360e2e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f635c0d2e20 con 0x7f635c0d29d0 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.194+0000 7f6360e2e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f635c0cea30 con 0x7f635c0d1270 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.194+0000 7f635ad9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f635c0d29d0 0x7f635c0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.194+0000 7f635a59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f635c161820 0x7f635c0d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50046/0 (socket says 172.21.15.161:50046) 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.194+0000 7f635a59c640 1 -- 172.21.15.161:0/1305082332 learned_addr learned my addr 172.21.15.161:0/1305082332 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.195+0000 7f6359d9b640 1 --2- 172.21.15.161:0/1305082332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d1270 0x7f635c0d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.195+0000 7f635ad9d640 1 -- 172.21.15.161:0/1305082332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d1270 msgr2=0x7f635c0d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.195+0000 7f635ad9d640 1 --2- 172.21.15.161:0/1305082332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d1270 0x7f635c0d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.195+0000 7f635ad9d640 1 -- 172.21.15.161:0/1305082332 >> v1:172.21.15.161:6789/0 conn(0x7f635c161820 legacy=0x7f635c0d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.195+0000 7f635ad9d640 1 -- 172.21.15.161:0/1305082332 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f635c0cfd70 con 0x7f635c0d29d0 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.195+0000 7f635959a640 1 -- 172.21.15.161:0/1305082332 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2750636537 0 0) 0x7f635c161c00 con 0x7f635c161820 2024-03-23T14:36:27.192 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f635ad9d640 1 --2- 172.21.15.161:0/1305082332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c0b0a80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63540097a0 tx=0x7f635402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1188ae5602eabc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f635959a640 1 -- 172.21.15.161:0/1305082332 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354031830 con 0x7f635c0d29d0 2024-03-23T14:36:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f635959a640 1 -- 172.21.15.161:0/1305082332 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6354031990 con 0x7f635c0d29d0 2024-03-23T14:36:27.193 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f635959a640 1 -- 172.21.15.161:0/1305082332 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6354031d20 con 0x7f635c0d29d0 2024-03-23T14:36:27.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f6360e2e640 1 -- 172.21.15.161:0/1305082332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 msgr2=0x7f635c0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f6360e2e640 1 --2- 172.21.15.161:0/1305082332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c0b0a80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f63540097a0 tx=0x7f635402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f6360e2e640 1 -- 172.21.15.161:0/1305082332 shutdown_connections 2024-03-23T14:36:27.197 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f6360e2e640 1 --2- 172.21.15.161:0/1305082332 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c0b0a80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f6360e2e640 1 --2- 172.21.15.161:0/1305082332 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c0d1270 0x7f635c0d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f6360e2e640 1 -- 172.21.15.161:0/1305082332 >> 172.21.15.161:0/1305082332 conn(0x7f635c09d190 msgr2=0x7f635c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f6360e2e640 1 -- 172.21.15.161:0/1305082332 shutdown_connections 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.196+0000 7f6360e2e640 1 -- 172.21.15.161:0/1305082332 wait complete. 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6360e2e640 1 Processor -- start 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6360e2e640 1 -- start start 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6360e2e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0d1270 0x7f635c150f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6360e2e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c06c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6360e2e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c161820 0x7f635c06c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6360e2e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f635c0cc8a0 con 0x7f635c0d29d0 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6360e2e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f635c14ae20 con 0x7f635c0d1270 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6360e2e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f635c0cc520 con 0x7f635c161820 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.198+0000 7f6359d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c06c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6359d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c06c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51688/0 (socket says 172.21.15.161:51688) 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6359d9b640 1 -- 172.21.15.161:0/3067193762 learned_addr learned my addr 172.21.15.161:0/3067193762 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f635a59c640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0d1270 0x7f635c150f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f635ad9d640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c161820 0x7f635c06c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6359d9b640 1 -- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c161820 msgr2=0x7f635c06c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6359d9b640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c161820 0x7f635c06c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6359d9b640 1 -- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0d1270 msgr2=0x7f635c150f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.199 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6359d9b640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0d1270 0x7f635c150f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6359d9b640 1 -- 172.21.15.161:0/3067193762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6354009400 con 0x7f635c0d29d0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6359d9b640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c06c2c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f635000e870 tx=0x7f635000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f634affd640 1 -- 172.21.15.161:0/3067193762 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63500094f0 con 0x7f635c0d29d0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f634affd640 1 -- 172.21.15.161:0/3067193762 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f635000c940 con 0x7f635c0d29d0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f635c14d620 con 0x7f635c0d29d0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f634affd640 1 -- 172.21.15.161:0/3067193762 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f635000ccf0 con 0x7f635c0d29d0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.199+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f635c14dbe0 con 0x7f635c0d29d0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.200+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f635c0ccb10 con 0x7f635c0d29d0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.201+0000 7f634affd640 1 -- 172.21.15.161:0/3067193762 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f6350013470 con 0x7f635c0d29d0 2024-03-23T14:36:27.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.201+0000 7f634affd640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f633c06b810 0x7f633c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.201+0000 7f634affd640 1 -- 172.21.15.161:0/3067193762 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f635008a2f0 con 0x7f635c0d29d0 2024-03-23T14:36:27.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.201+0000 7f635a59c640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f633c06b810 0x7f633c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.202+0000 7f635a59c640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f633c06b810 0x7f633c06dcd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f634c0012e0 tx=0x7f634c001700 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.205+0000 7f634affd640 1 -- 172.21.15.161:0/3067193762 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6350054cc0 con 0x7f635c0d29d0 2024-03-23T14:36:27.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.377+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f635c04f4e0 con 0x7f635c0d29d0 2024-03-23T14:36:27.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.378+0000 7f634affd640 1 -- 172.21.15.161:0/3067193762 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f6350058380 con 0x7f635c0d29d0 2024-03-23T14:36:27.374 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f633c06b810 msgr2=0x7f633c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f633c06b810 0x7f633c06dcd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f634c0012e0 tx=0x7f634c001700 comp rx=0 tx=0).stop 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 msgr2=0x7f635c06c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c06c2c0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f635000e870 tx=0x7f635000ed30 comp rx=0 tx=0).stop 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 shutdown_connections 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f633c06b810 0x7f633c06dcd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f635c161820 0x7f635c06c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f635c0d29d0 0x7f635c06c2c0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.376 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 --2- 172.21.15.161:0/3067193762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f635c0d1270 0x7f635c150f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 >> 172.21.15.161:0/3067193762 conn(0x7f635c09d190 msgr2=0x7f635c0d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:27.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 shutdown_connections 2024-03-23T14:36:27.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.380+0000 7f6360e2e640 1 -- 172.21.15.161:0/3067193762 wait complete. 2024-03-23T14:36:27.386 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:36:27.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.495+0000 7f9f66b63640 1 Processor -- start 2024-03-23T14:36:27.491 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.495+0000 7f9f66b63640 1 -- start start 2024-03-23T14:36:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f66b63640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9f60145520 0x7f9f60160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f66b63640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600b3a20 0x7f9f600d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f66b63640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f600d0880 con 0x7f9f600cfee0 2024-03-23T14:36:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f66b63640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9f601608c0 con 0x7f9f60145520 2024-03-23T14:36:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f66b63640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9f600ce410 con 0x7f9f600b3a20 2024-03-23T14:36:27.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f648ff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9f60145520 0x7f9f60160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f5ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9f600cfee0 0x7f9f60070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50056/0 (socket says 172.21.15.161:50056) 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f5ffff640 1 -- 172.21.15.161:0/2313531440 learned_addr learned my addr 172.21.15.161:0/2313531440 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f65100640 1 --2- 172.21.15.161:0/2313531440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600b3a20 0x7f9f600d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f648ff640 1 -- 172.21.15.161:0/2313531440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600b3a20 msgr2=0x7f9f600d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f648ff640 1 --2- 172.21.15.161:0/2313531440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600b3a20 0x7f9f600d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f648ff640 1 -- 172.21.15.161:0/2313531440 >> v1:172.21.15.161:6789/0 conn(0x7f9f600cfee0 legacy=0x7f9f60070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f648ff640 1 -- 172.21.15.161:0/2313531440 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f600ce6f0 con 0x7f9f60145520 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f5f7fe640 1 -- 172.21.15.161:0/2313531440 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1482509400 0 0) 0x7f9f600d0880 con 0x7f9f600cfee0 2024-03-23T14:36:27.494 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.496+0000 7f9f65100640 1 --2- 172.21.15.161:0/2313531440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600b3a20 0x7f9f600d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:27.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.497+0000 7f9f648ff640 1 --2- 172.21.15.161:0/2313531440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f60160380 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9f540097a0 tx=0x7f9f5402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56da9345748c8247 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.497+0000 7f9f5f7fe640 1 -- 172.21.15.161:0/2313531440 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f5402e070 con 0x7f9f60145520 2024-03-23T14:36:27.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.497+0000 7f9f5f7fe640 1 -- 172.21.15.161:0/2313531440 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f540315a0 con 0x7f9f60145520 2024-03-23T14:36:27.495 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.497+0000 7f9f5f7fe640 1 -- 172.21.15.161:0/2313531440 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f54031930 con 0x7f9f60145520 2024-03-23T14:36:27.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.498+0000 7f9f66b63640 1 -- 172.21.15.161:0/2313531440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 msgr2=0x7f9f60160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.498+0000 7f9f66b63640 1 --2- 172.21.15.161:0/2313531440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f60160380 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f9f540097a0 tx=0x7f9f5402b840 comp rx=0 tx=0).stop 2024-03-23T14:36:27.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.498+0000 7f9f66b63640 1 -- 172.21.15.161:0/2313531440 shutdown_connections 2024-03-23T14:36:27.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.498+0000 7f9f66b63640 1 --2- 172.21.15.161:0/2313531440 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600b3a20 0x7f9f600d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.498+0000 7f9f66b63640 1 --2- 172.21.15.161:0/2313531440 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f60160380 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.498+0000 7f9f66b63640 1 -- 172.21.15.161:0/2313531440 >> 172.21.15.161:0/2313531440 conn(0x7f9f6006b4a0 msgr2=0x7f9f6006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:27.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.498+0000 7f9f66b63640 1 -- 172.21.15.161:0/2313531440 shutdown_connections 2024-03-23T14:36:27.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.498+0000 7f9f66b63640 1 -- 172.21.15.161:0/2313531440 wait complete. 2024-03-23T14:36:27.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.499+0000 7f9f66b63640 1 Processor -- start 2024-03-23T14:36:27.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.499+0000 7f9f66b63640 1 -- start start 2024-03-23T14:36:27.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f66b63640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f600b3a20 0x7f9f60154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f66b63640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600cfee0 0x7f9f601563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f66b63640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f601f53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f66b63640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9f60155310 con 0x7f9f60145520 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f66b63640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9f601555f0 con 0x7f9f600b3a20 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f66b63640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9f601558d0 con 0x7f9f600cfee0 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f601f53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f601f53f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51712/0 (socket says 172.21.15.161:51712) 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 -- 172.21.15.161:0/3330886023 learned_addr learned my addr 172.21.15.161:0/3330886023 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f648ff640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f600b3a20 0x7f9f60154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f5ffff640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600cfee0 0x7f9f601563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 -- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600cfee0 msgr2=0x7f9f601563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600cfee0 0x7f9f601563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 -- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f600b3a20 msgr2=0x7f9f60154ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f600b3a20 0x7f9f60154ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 -- 172.21.15.161:0/3330886023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f58009a70 con 0x7f9f60145520 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.500+0000 7f9f65100640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f601f53f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800efb0 tx=0x7f9f5800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.501+0000 7f9f5d7fa640 1 -- 172.21.15.161:0/3330886023 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f58009d50 con 0x7f9f60145520 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.501+0000 7f9f5d7fa640 1 -- 172.21.15.161:0/3330886023 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f5800f040 con 0x7f9f60145520 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.501+0000 7f9f5d7fa640 1 -- 172.21.15.161:0/3330886023 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f580146a0 con 0x7f9f60145520 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.501+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f54009400 con 0x7f9f60145520 2024-03-23T14:36:27.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.501+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f60155e90 con 0x7f9f60145520 2024-03-23T14:36:27.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.502+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f600cec70 con 0x7f9f60145520 2024-03-23T14:36:27.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.502+0000 7f9f5d7fa640 1 -- 172.21.15.161:0/3330886023 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9f58014800 con 0x7f9f60145520 2024-03-23T14:36:27.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.502+0000 7f9f5d7fa640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f4406b810 0x7f9f4406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.502+0000 7f9f5d7fa640 1 -- 172.21.15.161:0/3330886023 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f9f5808b270 con 0x7f9f60145520 2024-03-23T14:36:27.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.502+0000 7f9f648ff640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f4406b810 0x7f9f4406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.503+0000 7f9f648ff640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f4406b810 0x7f9f4406dcd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9f54036000 tx=0x7f9f5403d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.507+0000 7f9f5d7fa640 1 -- 172.21.15.161:0/3330886023 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9f58055b20 con 0x7f9f60145520 2024-03-23T14:36:27.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.677+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f9f601560b0 con 0x7f9f60145520 2024-03-23T14:36:27.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.677+0000 7f9f5d7fa640 1 -- 172.21.15.161:0/3330886023 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) ==== 104+0+11 (secure 0 0 0) 0x7f9f58018020 con 0x7f9f60145520 2024-03-23T14:36:27.673 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:27.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f4406b810 msgr2=0x7f9f4406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f4406b810 0x7f9f4406dcd0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9f54036000 tx=0x7f9f5403d040 comp rx=0 tx=0).stop 2024-03-23T14:36:27.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 msgr2=0x7f9f601f53f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f601f53f0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f9f5800efb0 tx=0x7f9f5800c490 comp rx=0 tx=0).stop 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 shutdown_connections 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f4406b810 0x7f9f4406dcd0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f60145520 0x7f9f601f53f0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f600cfee0 0x7f9f601563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 --2- 172.21.15.161:0/3330886023 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f600b3a20 0x7f9f60154ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.679+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 >> 172.21.15.161:0/3330886023 conn(0x7f9f6006b4a0 msgr2=0x7f9f600b09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.680+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 shutdown_connections 2024-03-23T14:36:27.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.680+0000 7f9f66b63640 1 -- 172.21.15.161:0/3330886023 wait complete. 2024-03-23T14:36:27.687 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:36:27.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.795+0000 7f61bc007640 1 Processor -- start 2024-03-23T14:36:27.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.795+0000 7f61bc007640 1 -- start start 2024-03-23T14:36:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.795+0000 7f61bc007640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f61b41552f0 0x7f61b40b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.795+0000 7f61bc007640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f61b40d0380 0x7f61b406ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.795+0000 7f61bc007640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61b415f480 con 0x7f61b40b2aa0 2024-03-23T14:36:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.795+0000 7f61bc007640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f61b41556d0 con 0x7f61b40d0380 2024-03-23T14:36:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.795+0000 7f61bc007640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f61b415e870 con 0x7f61b41552f0 2024-03-23T14:36:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.796+0000 7f61ba5a4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f61b40d0380 0x7f61b406ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.794 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.796+0000 7f61ba5a4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f61b40d0380 0x7f61b406ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51718/0 (socket says 172.21.15.161:51718) 2024-03-23T14:36:27.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.796+0000 7f61ba5a4640 1 -- 172.21.15.161:0/3276392869 learned_addr learned my addr 172.21.15.161:0/3276392869 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:27.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.796+0000 7f61b9da3640 1 --2- 172.21.15.161:0/3276392869 >> v2:172.21.15.183:3300/0 conn(0x7f61b41552f0 0x7f61b40b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.796+0000 7f61ba5a4640 1 -- 172.21.15.161:0/3276392869 >> v2:172.21.15.183:3300/0 conn(0x7f61b41552f0 msgr2=0x7f61b40b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:27.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.796+0000 7f61ba5a4640 1 --2- 172.21.15.161:0/3276392869 >> v2:172.21.15.183:3300/0 conn(0x7f61b41552f0 0x7f61b40b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.796+0000 7f61ba5a4640 1 -- 172.21.15.161:0/3276392869 >> v1:172.21.15.183:6789/0 conn(0x7f61b40b2aa0 legacy=0x7f61b40cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.796+0000 7f61ba5a4640 1 -- 172.21.15.161:0/3276392869 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61b415eb50 con 0x7f61b40d0380 2024-03-23T14:36:27.795 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61ba5a4640 1 --2- 172.21.15.161:0/3276392869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40d0380 0x7f61b406ded0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f61b406d4d0 tx=0x7f61ac02e180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21c223e09a7ae652 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61b8da1640 1 -- 172.21.15.161:0/3276392869 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac00b0d0 con 0x7f61b40d0380 2024-03-23T14:36:27.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61b8da1640 1 -- 172.21.15.161:0/3276392869 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61ac00b230 con 0x7f61b40d0380 2024-03-23T14:36:27.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61b8da1640 1 -- 172.21.15.161:0/3276392869 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61ac009600 con 0x7f61b40d0380 2024-03-23T14:36:27.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61bc007640 1 -- 172.21.15.161:0/3276392869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40d0380 msgr2=0x7f61b406ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61bc007640 1 --2- 172.21.15.161:0/3276392869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40d0380 0x7f61b406ded0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f61b406d4d0 tx=0x7f61ac02e180 comp rx=0 tx=0).stop 2024-03-23T14:36:27.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61bc007640 1 -- 172.21.15.161:0/3276392869 shutdown_connections 2024-03-23T14:36:27.796 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61bc007640 1 --2- 172.21.15.161:0/3276392869 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40d0380 0x7f61b406ded0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61bc007640 1 --2- 172.21.15.161:0/3276392869 >> v2:172.21.15.183:3300/0 conn(0x7f61b41552f0 0x7f61b40b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61bc007640 1 -- 172.21.15.161:0/3276392869 >> 172.21.15.161:0/3276392869 conn(0x7f61b406b260 msgr2=0x7f61b406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61bc007640 1 -- 172.21.15.161:0/3276392869 shutdown_connections 2024-03-23T14:36:27.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.797+0000 7f61bc007640 1 -- 172.21.15.161:0/3276392869 wait complete. 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61bc007640 1 Processor -- start 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61bc007640 1 -- start start 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61bc007640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40b2aa0 0x7f61b415e060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61bc007640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61b40d0380 0x7f61b4159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61bc007640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61b41552f0 0x7f61b413d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61bc007640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f61b40cd0c0 con 0x7f61b40b2aa0 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61bc007640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f61b415e5a0 con 0x7f61b41552f0 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61bc007640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f61b415c8e0 con 0x7f61b40d0380 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61b9da3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40b2aa0 0x7f61b415e060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61b9da3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40b2aa0 0x7f61b415e060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51722/0 (socket says 172.21.15.161:51722) 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61b9da3640 1 -- 172.21.15.161:0/228912637 learned_addr learned my addr 172.21.15.161:0/228912637 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61ba5a4640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61b41552f0 0x7f61b413d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.799+0000 7f61b95a2640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61b40d0380 0x7f61b4159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61b9da3640 1 -- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61b40d0380 msgr2=0x7f61b4159550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61b9da3640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61b40d0380 0x7f61b4159550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61b9da3640 1 -- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61b41552f0 msgr2=0x7f61b413d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61b9da3640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61b41552f0 0x7f61b413d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61b9da3640 1 -- 172.21.15.161:0/228912637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61ac00bd40 con 0x7f61b40b2aa0 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61b9da3640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40b2aa0 0x7f61b415e060 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f61a400e870 tx=0x7f61a400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61aa7fc640 1 -- 172.21.15.161:0/228912637 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a40094f0 con 0x7f61b40b2aa0 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61aa7fc640 1 -- 172.21.15.161:0/228912637 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f61a400c940 con 0x7f61b40b2aa0 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61b415cbc0 con 0x7f61b40b2aa0 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61aa7fc640 1 -- 172.21.15.161:0/228912637 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f61a400ccf0 con 0x7f61b40b2aa0 2024-03-23T14:36:27.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.800+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61b415d180 con 0x7f61b40b2aa0 2024-03-23T14:36:27.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.801+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61b404f4e0 con 0x7f61b40b2aa0 2024-03-23T14:36:27.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.802+0000 7f61aa7fc640 1 -- 172.21.15.161:0/228912637 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f61a4013470 con 0x7f61b40b2aa0 2024-03-23T14:36:27.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.802+0000 7f61aa7fc640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f619806b810 0x7f619806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:27.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.802+0000 7f61aa7fc640 1 -- 172.21.15.161:0/228912637 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f61a408a400 con 0x7f61b40b2aa0 2024-03-23T14:36:27.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.802+0000 7f61b95a2640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f619806b810 0x7f619806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:27.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.802+0000 7f61b95a2640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f619806b810 0x7f619806dcd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f61b4153f70 tx=0x7f61b0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:27.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.806+0000 7f61aa7fc640 1 -- 172.21.15.161:0/228912637 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f61a4054d40 con 0x7f61b40b2aa0 2024-03-23T14:36:27.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.978+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f61b414f120 con 0x7f61b40b2aa0 2024-03-23T14:36:27.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.979+0000 7f61aa7fc640 1 -- 172.21.15.161:0/228912637 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) ==== 100+0+11 (secure 0 0 0) 0x7f61a4058400 con 0x7f61b40b2aa0 2024-03-23T14:36:27.975 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:36:27.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f619806b810 msgr2=0x7f619806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f619806b810 0x7f619806dcd0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f61b4153f70 tx=0x7f61b0009040 comp rx=0 tx=0).stop 2024-03-23T14:36:27.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40b2aa0 msgr2=0x7f61b415e060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:27.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40b2aa0 0x7f61b415e060 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f61a400e870 tx=0x7f61a400ed30 comp rx=0 tx=0).stop 2024-03-23T14:36:27.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 shutdown_connections 2024-03-23T14:36:27.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f619806b810 0x7f619806dcd0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f61b41552f0 0x7f61b413d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f61b40d0380 0x7f61b4159550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 --2- 172.21.15.161:0/228912637 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f61b40b2aa0 0x7f61b415e060 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:27.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 >> 172.21.15.161:0/228912637 conn(0x7f61b406b260 msgr2=0x7f61b4153540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:27.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 shutdown_connections 2024-03-23T14:36:27.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:27.981+0000 7f61bc007640 1 -- 172.21.15.161:0/228912637 wait complete. 2024-03-23T14:36:27.987 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-23T14:36:28.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.096+0000 7fb5a9119640 1 Processor -- start 2024-03-23T14:36:28.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.096+0000 7fb5a9119640 1 -- start start 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a9119640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5a4145520 0x7fb5a4160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a9119640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 0x7fb5a40d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a9119640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a40d0880 con 0x7fb5a40cfee0 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a9119640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb5a41608c0 con 0x7fb5a4145520 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a9119640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a40ce410 con 0x7fb5a40b3a20 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5a4145520 0x7fb5a4160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb59bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb5a40cfee0 0x7fb5a4070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50070/0 (socket says 172.21.15.161:50070) 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb59bfff640 1 -- 172.21.15.161:0/3344478122 learned_addr learned my addr 172.21.15.161:0/3344478122 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:28.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a359e640 1 --2- 172.21.15.161:0/3344478122 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 0x7fb5a40d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a2d9d640 1 -- 172.21.15.161:0/3344478122 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 msgr2=0x7fb5a40d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a2d9d640 1 --2- 172.21.15.161:0/3344478122 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 0x7fb5a40d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a2d9d640 1 -- 172.21.15.161:0/3344478122 >> v1:172.21.15.161:6789/0 conn(0x7fb5a40cfee0 legacy=0x7fb5a4070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a2d9d640 1 -- 172.21.15.161:0/3344478122 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a40ce6f0 con 0x7fb5a4145520 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.097+0000 7fb5a259c640 1 -- 172.21.15.161:0/3344478122 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 186629496 0 0) 0x7fb5a40d0880 con 0x7fb5a40cfee0 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.098+0000 7fb5a2d9d640 1 --2- 172.21.15.161:0/3344478122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a4160380 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb5940097a0 tx=0x7fb59402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d00871007459035 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.098+0000 7fb5a259c640 1 -- 172.21.15.161:0/3344478122 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59402e070 con 0x7fb5a4145520 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.098+0000 7fb5a259c640 1 -- 172.21.15.161:0/3344478122 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5940315a0 con 0x7fb5a4145520 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.098+0000 7fb5a259c640 1 -- 172.21.15.161:0/3344478122 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb594031930 con 0x7fb5a4145520 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.098+0000 7fb5a9119640 1 -- 172.21.15.161:0/3344478122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 msgr2=0x7fb5a4160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.099+0000 7fb5a9119640 1 --2- 172.21.15.161:0/3344478122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a4160380 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb5940097a0 tx=0x7fb59402b840 comp rx=0 tx=0).stop 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.099+0000 7fb5a9119640 1 -- 172.21.15.161:0/3344478122 shutdown_connections 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.099+0000 7fb5a9119640 1 --2- 172.21.15.161:0/3344478122 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 0x7fb5a40d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.099+0000 7fb5a9119640 1 --2- 172.21.15.161:0/3344478122 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a4160380 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.099+0000 7fb5a9119640 1 -- 172.21.15.161:0/3344478122 >> 172.21.15.161:0/3344478122 conn(0x7fb5a406b4a0 msgr2=0x7fb5a406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.099+0000 7fb5a9119640 1 -- 172.21.15.161:0/3344478122 shutdown_connections 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.099+0000 7fb5a9119640 1 -- 172.21.15.161:0/3344478122 wait complete. 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.100+0000 7fb5a9119640 1 Processor -- start 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.100+0000 7fb5a9119640 1 -- start start 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a9119640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 0x7fb5a413ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a9119640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a40cfee0 0x7fb5a41f20e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a9119640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a415fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a9119640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a4154a50 con 0x7fb5a4145520 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a9119640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a4154d30 con 0x7fb5a40cfee0 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a9119640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a4153890 con 0x7fb5a40b3a20 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a415fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a415fcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51744/0 (socket says 172.21.15.161:51744) 2024-03-23T14:36:28.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 -- 172.21.15.161:0/938104626 learned_addr learned my addr 172.21.15.161:0/938104626 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a2d9d640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 0x7fb5a413ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb59bfff640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a40cfee0 0x7fb5a41f20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 -- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 msgr2=0x7fb5a413ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 0x7fb5a413ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 -- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a40cfee0 msgr2=0x7fb5a41f20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a40cfee0 0x7fb5a41f20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 -- 172.21.15.161:0/938104626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb594009400 con 0x7fb5a4145520 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a359e640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a415fcd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009e30 tx=0x7fb59c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb59b7fe640 1 -- 172.21.15.161:0/938104626 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c009ca0 con 0x7fb5a4145520 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb59b7fe640 1 -- 172.21.15.161:0/938104626 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb59c00f040 con 0x7fb5a4145520 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a4153b70 con 0x7fb5a4145520 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.101+0000 7fb59b7fe640 1 -- 172.21.15.161:0/938104626 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb59c0146c0 con 0x7fb5a4145520 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.102+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a41534b0 con 0x7fb5a4145520 2024-03-23T14:36:28.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.102+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a4073570 con 0x7fb5a4145520 2024-03-23T14:36:28.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.103+0000 7fb59b7fe640 1 -- 172.21.15.161:0/938104626 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb59c014820 con 0x7fb5a4145520 2024-03-23T14:36:28.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.103+0000 7fb59b7fe640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb58406b810 0x7fb58406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.103+0000 7fb59b7fe640 1 -- 172.21.15.161:0/938104626 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fb59c08b100 con 0x7fb5a4145520 2024-03-23T14:36:28.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.103+0000 7fb5a2d9d640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb58406b810 0x7fb58406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.104+0000 7fb5a2d9d640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb58406b810 0x7fb58406dcd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb594036000 tx=0x7fb59403e040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.109+0000 7fb59b7fe640 1 -- 172.21.15.161:0/938104626 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb59c055ad0 con 0x7fb5a4145520 2024-03-23T14:36:28.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.278+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fb5a4154e70 con 0x7fb5a4145520 2024-03-23T14:36:28.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.278+0000 7fb59b7fe640 1 -- 172.21.15.161:0/938104626 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v36) ==== 97+0+11 (secure 0 0 0) 0x7fb59c059190 con 0x7fb5a4145520 2024-03-23T14:36:28.275 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:28.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb58406b810 msgr2=0x7fb58406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb58406b810 0x7fb58406dcd0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb594036000 tx=0x7fb59403e040 comp rx=0 tx=0).stop 2024-03-23T14:36:28.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 msgr2=0x7fb5a415fcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a415fcd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb59c009e30 tx=0x7fb59c00c490 comp rx=0 tx=0).stop 2024-03-23T14:36:28.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 shutdown_connections 2024-03-23T14:36:28.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb58406b810 0x7fb58406dcd0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a4145520 0x7fb5a415fcd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a40cfee0 0x7fb5a41f20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 --2- 172.21.15.161:0/938104626 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a40b3a20 0x7fb5a413ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 >> 172.21.15.161:0/938104626 conn(0x7fb5a406b4a0 msgr2=0x7fb5a40afee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:28.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 shutdown_connections 2024-03-23T14:36:28.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.280+0000 7fb5a9119640 1 -- 172.21.15.161:0/938104626 wait complete. 2024-03-23T14:36:28.287 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:36:28.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.393+0000 7f8dacdc7640 1 Processor -- start 2024-03-23T14:36:28.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.393+0000 7f8dacdc7640 1 -- start start 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8dacdc7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8da8145080 0x7f8da8072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8dacdc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da81607d0 0x7f8da815fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8dacdc7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8da806cbf0 0x7f8da8160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8dacdc7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8da8071010 con 0x7f8da8145080 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8dacdc7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f8da80cf4f0 con 0x7f8da806cbf0 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8dacdc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8da80cef30 con 0x7f8da81607d0 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8da8145080 0x7f8da8072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8da8145080 0x7f8da8072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51758/0 (socket says 172.21.15.161:51758) 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da659c640 1 -- 172.21.15.161:0/309042631 learned_addr learned my addr 172.21.15.161:0/309042631 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8d9ffff640 1 --2- 172.21.15.161:0/309042631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da81607d0 0x7f8da815fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da6d9d640 1 --2- 172.21.15.161:0/309042631 >> v2:172.21.15.183:3300/0 conn(0x7f8da806cbf0 0x7f8da8160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da659c640 1 -- 172.21.15.161:0/309042631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da81607d0 msgr2=0x7f8da815fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da659c640 1 --2- 172.21.15.161:0/309042631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da81607d0 0x7f8da815fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da659c640 1 -- 172.21.15.161:0/309042631 >> v2:172.21.15.183:3300/0 conn(0x7f8da806cbf0 msgr2=0x7f8da8160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da659c640 1 --2- 172.21.15.161:0/309042631 >> v2:172.21.15.183:3300/0 conn(0x7f8da806cbf0 0x7f8da8160020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.394+0000 7f8da659c640 1 -- 172.21.15.161:0/309042631 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da80cf210 con 0x7f8da8145080 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.395+0000 7f8da659c640 1 --2- 172.21.15.161:0/309042631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da8145080 0x7f8da8072a00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8da80734c0 tx=0x7f8d9802b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c9e488da4e3bd2f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.395+0000 7f8da5d9b640 1 -- 172.21.15.161:0/309042631 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d980317e0 con 0x7f8da8145080 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.395+0000 7f8da5d9b640 1 -- 172.21.15.161:0/309042631 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d98031940 con 0x7f8da8145080 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.395+0000 7f8da5d9b640 1 -- 172.21.15.161:0/309042631 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d98031cd0 con 0x7f8da8145080 2024-03-23T14:36:28.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 -- 172.21.15.161:0/309042631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da8145080 msgr2=0x7f8da8072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/309042631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da8145080 0x7f8da8072a00 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f8da80734c0 tx=0x7f8d9802b880 comp rx=0 tx=0).stop 2024-03-23T14:36:28.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 -- 172.21.15.161:0/309042631 shutdown_connections 2024-03-23T14:36:28.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/309042631 >> v2:172.21.15.183:3300/0 conn(0x7f8da806cbf0 0x7f8da8160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/309042631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da81607d0 0x7f8da815fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/309042631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da8145080 0x7f8da8072a00 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 -- 172.21.15.161:0/309042631 >> 172.21.15.161:0/309042631 conn(0x7f8da80b17f0 msgr2=0x7f8da80b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:28.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 -- 172.21.15.161:0/309042631 shutdown_connections 2024-03-23T14:36:28.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.396+0000 7f8dacdc7640 1 -- 172.21.15.161:0/309042631 wait complete. 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.397+0000 7f8dacdc7640 1 Processor -- start 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.397+0000 7f8dacdc7640 1 -- start start 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8dacdc7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da806cbf0 0x7f8da81f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8dacdc7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8da8145080 0x7f8da81f51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8dacdc7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da81607d0 0x7f8da813b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8dacdc7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8da81f6070 con 0x7f8da81607d0 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8dacdc7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8da81f5ab0 con 0x7f8da8145080 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8dacdc7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8da81f5d90 con 0x7f8da806cbf0 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da81607d0 0x7f8da813b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da81607d0 0x7f8da813b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51766/0 (socket says 172.21.15.161:51766) 2024-03-23T14:36:28.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 -- 172.21.15.161:0/3358353009 learned_addr learned my addr 172.21.15.161:0/3358353009 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da659c640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da806cbf0 0x7f8da81f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8d9ffff640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8da8145080 0x7f8da81f51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 -- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da806cbf0 msgr2=0x7f8da81f2560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da806cbf0 0x7f8da81f2560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 -- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8da8145080 msgr2=0x7f8da81f51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8da8145080 0x7f8da81f51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 -- 172.21.15.161:0/3358353009 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d980094d0 con 0x7f8da81607d0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8da6d9d640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da81607d0 0x7f8da813b8e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8da000eed0 tx=0x7f8da000b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8d9effd640 1 -- 172.21.15.161:0/3358353009 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0010040 con 0x7f8da81607d0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.398+0000 7f8d9effd640 1 -- 172.21.15.161:0/3358353009 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da000bd20 con 0x7f8da81607d0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.399+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da81f1d60 con 0x7f8da81607d0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.399+0000 7f8d9effd640 1 -- 172.21.15.161:0/3358353009 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0014620 con 0x7f8da81607d0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.399+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da81f2aa0 con 0x7f8da81607d0 2024-03-23T14:36:28.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.400+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da81f1bc0 con 0x7f8da81607d0 2024-03-23T14:36:28.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.400+0000 7f8d9effd640 1 -- 172.21.15.161:0/3358353009 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8da0014780 con 0x7f8da81607d0 2024-03-23T14:36:28.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.400+0000 7f8d9effd640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8d8806b810 0x7f8d8806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.400+0000 7f8d9effd640 1 -- 172.21.15.161:0/3358353009 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f8da008abf0 con 0x7f8da81607d0 2024-03-23T14:36:28.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.401+0000 7f8da659c640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8d8806b810 0x7f8d8806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.401+0000 7f8da659c640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8d8806b810 0x7f8d8806dcd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8d98036000 tx=0x7f8d9803d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.405+0000 7f8d9effd640 1 -- 172.21.15.161:0/3358353009 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8da81f1bc0 con 0x7f8da81607d0 2024-03-23T14:36:28.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.586+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8da81f2280 con 0x7f8da81607d0 2024-03-23T14:36:28.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.586+0000 7f8d9effd640 1 -- 172.21.15.161:0/3358353009 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) ==== 80+0+0 (secure 0 0 0) 0x7f8da81f2280 con 0x7f8da81607d0 2024-03-23T14:36:28.583 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:36:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8d8806b810 msgr2=0x7f8d8806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8d8806b810 0x7f8d8806dcd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f8d98036000 tx=0x7f8d9803d040 comp rx=0 tx=0).stop 2024-03-23T14:36:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da81607d0 msgr2=0x7f8da813b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da81607d0 0x7f8da813b8e0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f8da000eed0 tx=0x7f8da000b480 comp rx=0 tx=0).stop 2024-03-23T14:36:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 shutdown_connections 2024-03-23T14:36:28.585 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8d8806b810 0x7f8d8806dcd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8da81607d0 0x7f8da813b8e0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8da8145080 0x7f8da81f51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 --2- 172.21.15.161:0/3358353009 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8da806cbf0 0x7f8da81f2560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 >> 172.21.15.161:0/3358353009 conn(0x7f8da80b17f0 msgr2=0x7f8da815f8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:28.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 shutdown_connections 2024-03-23T14:36:28.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.588+0000 7f8dacdc7640 1 -- 172.21.15.161:0/3358353009 wait complete. 2024-03-23T14:36:28.596 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:28.596 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:28.612 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:28.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.756+0000 7f1ef353e640 1 Processor -- start 2024-03-23T14:36:28.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.756+0000 7f1ef353e640 1 -- start start 2024-03-23T14:36:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.756+0000 7f1ef353e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec06eca0 0x7f1eec165d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.756+0000 7f1ef353e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eec15be70 con 0x7f1eec0724e0 2024-03-23T14:36:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.756+0000 7f1ef353e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eec15c150 con 0x7f1eec072100 2024-03-23T14:36:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.756+0000 7f1ef353e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec15c430 con 0x7f1eec06eca0 2024-03-23T14:36:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1ef0ad9640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f1eec0724e0 0x7f1eec06e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50074/0 (socket says 172.21.15.161:50074) 2024-03-23T14:36:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1ef0ad9640 1 -- 172.21.15.161:0/119579906 learned_addr learned my addr 172.21.15.161:0/119579906 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:28.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1ef12da640 1 --1- 172.21.15.161:0/119579906 >> v1:172.21.15.183:6789/0 conn(0x7f1eec072100 0x7f1eec15b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:43210/0 (socket says 172.21.15.161:43210) 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1ef1adb640 1 --2- 172.21.15.161:0/119579906 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec06eca0 0x7f1eec165d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1edffff640 1 -- 172.21.15.161:0/119579906 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2654374715 0 0) 0x7f1eec15be70 con 0x7f1eec0724e0 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1edffff640 1 -- 172.21.15.161:0/119579906 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1ed8003520 con 0x7f1eec0724e0 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1edffff640 1 -- 172.21.15.161:0/119579906 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1254003206 0 0) 0x7f1eec15c150 con 0x7f1eec072100 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1edffff640 1 -- 172.21.15.161:0/119579906 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eec15be70 con 0x7f1eec072100 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1ef1adb640 1 -- 172.21.15.161:0/119579906 >> v1:172.21.15.183:6789/0 conn(0x7f1eec072100 legacy=0x7f1eec15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1ef1adb640 1 -- 172.21.15.161:0/119579906 >> v1:172.21.15.161:6789/0 conn(0x7f1eec0724e0 legacy=0x7f1eec06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.757+0000 7f1ef1adb640 1 -- 172.21.15.161:0/119579906 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec0b2c70 con 0x7f1eec06eca0 2024-03-23T14:36:28.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1ef1adb640 1 --2- 172.21.15.161:0/119579906 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec06eca0 0x7f1eec165d70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009dc0 tx=0x7f1ee402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26eea3d3ded140de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1edffff640 1 -- 172.21.15.161:0/119579906 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee402ccd0 con 0x7f1eec06eca0 2024-03-23T14:36:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1edffff640 1 -- 172.21.15.161:0/119579906 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ee402ce30 con 0x7f1eec06eca0 2024-03-23T14:36:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1edffff640 1 -- 172.21.15.161:0/119579906 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4034640 con 0x7f1eec06eca0 2024-03-23T14:36:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1ef353e640 1 -- 172.21.15.161:0/119579906 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec06eca0 msgr2=0x7f1eec165d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1ef353e640 1 --2- 172.21.15.161:0/119579906 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec06eca0 0x7f1eec165d70 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1ee4009dc0 tx=0x7f1ee402bf20 comp rx=0 tx=0).stop 2024-03-23T14:36:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1ef353e640 1 -- 172.21.15.161:0/119579906 shutdown_connections 2024-03-23T14:36:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1ef353e640 1 --2- 172.21.15.161:0/119579906 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec06eca0 0x7f1eec165d70 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.758+0000 7f1ef353e640 1 -- 172.21.15.161:0/119579906 >> 172.21.15.161:0/119579906 conn(0x7f1eec0b17f0 msgr2=0x7f1eec0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.759+0000 7f1ef353e640 1 -- 172.21.15.161:0/119579906 shutdown_connections 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.759+0000 7f1ef353e640 1 -- 172.21.15.161:0/119579906 wait complete. 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef353e640 1 Processor -- start 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef353e640 1 -- start start 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef353e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1eec06eca0 0x7f1eec148520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef353e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eec072100 0x7f1eec0a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef353e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec0724e0 0x7f1eec0ce3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef353e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec0cd0e0 con 0x7f1eec06eca0 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef353e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec0cd330 con 0x7f1eec072100 2024-03-23T14:36:28.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef353e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1eec0cd610 con 0x7f1eec0724e0 2024-03-23T14:36:28.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef12da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1eec06eca0 0x7f1eec148520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef12da640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1eec06eca0 0x7f1eec148520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51776/0 (socket says 172.21.15.161:51776) 2024-03-23T14:36:28.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef12da640 1 -- 172.21.15.161:0/1814899783 learned_addr learned my addr 172.21.15.161:0/1814899783 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:28.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef0ad9640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eec072100 0x7f1eec0a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.760+0000 7f1ef1adb640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec0724e0 0x7f1eec0ce3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1ef12da640 1 -- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec0724e0 msgr2=0x7f1eec0ce3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1ef12da640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec0724e0 0x7f1eec0ce3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1ef12da640 1 -- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eec072100 msgr2=0x7f1eec0a94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1ef12da640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eec072100 0x7f1eec0a94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1ef12da640 1 -- 172.21.15.161:0/1814899783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ee4009a70 con 0x7f1eec06eca0 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1ef12da640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1eec06eca0 0x7f1eec148520 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1ee000c430 tx=0x7f1ee000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1eddffb640 1 -- 172.21.15.161:0/1814899783 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee00074e0 con 0x7f1eec06eca0 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1eddffb640 1 -- 172.21.15.161:0/1814899783 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ee0007660 con 0x7f1eec06eca0 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eec0cd8f0 con 0x7f1eec06eca0 2024-03-23T14:36:28.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1eddffb640 1 -- 172.21.15.161:0/1814899783 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee0004a20 con 0x7f1eec06eca0 2024-03-23T14:36:28.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.761+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eec0cdeb0 con 0x7f1eec06eca0 2024-03-23T14:36:28.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.762+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eec0735e0 con 0x7f1eec06eca0 2024-03-23T14:36:28.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.763+0000 7f1eddffb640 1 -- 172.21.15.161:0/1814899783 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1ee0004b80 con 0x7f1eec06eca0 2024-03-23T14:36:28.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.763+0000 7f1eddffb640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1ed006b810 0x7f1ed006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:28.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.763+0000 7f1eddffb640 1 -- 172.21.15.161:0/1814899783 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f1ee0089cd0 con 0x7f1eec06eca0 2024-03-23T14:36:28.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.763+0000 7f1ef0ad9640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1ed006b810 0x7f1ed006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:28.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.764+0000 7f1ef0ad9640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1ed006b810 0x7f1ed006dcd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1eec1551e0 tx=0x7f1ee8007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:28.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.768+0000 7f1eddffb640 1 -- 172.21.15.161:0/1814899783 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1ee0058200 con 0x7f1eec06eca0 2024-03-23T14:36:28.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.939+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1eec04f550 con 0x7f1eec06eca0 2024-03-23T14:36:28.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.940+0000 7f1eddffb640 1 -- 172.21.15.161:0/1814899783 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7f1ee0057d90 con 0x7f1eec06eca0 2024-03-23T14:36:28.937 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:28.937 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:28.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1ed006b810 msgr2=0x7f1ed006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1ed006b810 0x7f1ed006dcd0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f1eec1551e0 tx=0x7f1ee8007d30 comp rx=0 tx=0).stop 2024-03-23T14:36:28.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1eec06eca0 msgr2=0x7f1eec148520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1eec06eca0 0x7f1eec148520 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1ee000c430 tx=0x7f1ee000c8f0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 shutdown_connections 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1ed006b810 0x7f1ed006dcd0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1eec0724e0 0x7f1eec0ce3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1eec072100 0x7f1eec0a94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 --2- 172.21.15.161:0/1814899783 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1eec06eca0 0x7f1eec148520 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 >> 172.21.15.161:0/1814899783 conn(0x7f1eec0b17f0 msgr2=0x7f1eec0d0250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 shutdown_connections 2024-03-23T14:36:28.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:28.942+0000 7f1ef353e640 1 -- 172.21.15.161:0/1814899783 wait complete. 2024-03-23T14:36:28.950 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:04.688180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:05.241774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-23T14:36:18.738957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:28.950 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:29.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.058+0000 7f3f2a95c640 1 Processor -- start 2024-03-23T14:36:29.054 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.058+0000 7f3f2a95c640 1 -- start start 2024-03-23T14:36:29.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f2a95c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3f24145570 0x7f3f240cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f2a95c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3f240d0380 0x7f3f240b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f2a95c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f240cf200 con 0x7f3f2415cdf0 2024-03-23T14:36:29.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f2a95c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3f2415d2a0 con 0x7f3f24145570 2024-03-23T14:36:29.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f2a95c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f3f240ce5f0 con 0x7f3f240d0380 2024-03-23T14:36:29.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f23fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3f24145570 0x7f3f240cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f23fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3f24145570 0x7f3f240cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51792/0 (socket says 172.21.15.161:51792) 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f23fff640 1 -- 172.21.15.161:0/611803910 learned_addr learned my addr 172.21.15.161:0/611803910 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f28ef9640 1 --2- 172.21.15.161:0/611803910 >> v2:172.21.15.183:3300/0 conn(0x7f3f240d0380 0x7f3f240b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f23fff640 1 -- 172.21.15.161:0/611803910 >> v2:172.21.15.183:3300/0 conn(0x7f3f240d0380 msgr2=0x7f3f240b2b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f23fff640 1 --2- 172.21.15.161:0/611803910 >> v2:172.21.15.183:3300/0 conn(0x7f3f240d0380 0x7f3f240b2b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f23fff640 1 -- 172.21.15.161:0/611803910 >> v1:172.21.15.183:6789/0 conn(0x7f3f2415cdf0 legacy=0x7f3f240cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.059+0000 7f3f23fff640 1 -- 172.21.15.161:0/611803910 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f240ce8d0 con 0x7f3f24145570 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.060+0000 7f3f23fff640 1 --2- 172.21.15.161:0/611803910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f24145570 0x7f3f240cdf30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3f240b5060 tx=0x7f3f1402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d65a93144f756dca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.060+0000 7f3f22ffd640 1 -- 172.21.15.161:0/611803910 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f14030860 con 0x7f3f24145570 2024-03-23T14:36:29.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.060+0000 7f3f22ffd640 1 -- 172.21.15.161:0/611803910 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f140309c0 con 0x7f3f24145570 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.060+0000 7f3f22ffd640 1 -- 172.21.15.161:0/611803910 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f14030d50 con 0x7f3f24145570 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.060+0000 7f3f2a95c640 1 -- 172.21.15.161:0/611803910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f24145570 msgr2=0x7f3f240cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.060+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/611803910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f24145570 0x7f3f240cdf30 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f3f240b5060 tx=0x7f3f1402b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.061+0000 7f3f2a95c640 1 -- 172.21.15.161:0/611803910 shutdown_connections 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.061+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/611803910 >> v2:172.21.15.183:3300/0 conn(0x7f3f240d0380 0x7f3f240b2b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.061+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/611803910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f24145570 0x7f3f240cdf30 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.061+0000 7f3f2a95c640 1 -- 172.21.15.161:0/611803910 >> 172.21.15.161:0/611803910 conn(0x7f3f2406b260 msgr2=0x7f3f2406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.061+0000 7f3f2a95c640 1 -- 172.21.15.161:0/611803910 shutdown_connections 2024-03-23T14:36:29.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.061+0000 7f3f2a95c640 1 -- 172.21.15.161:0/611803910 wait complete. 2024-03-23T14:36:29.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.062+0000 7f3f2a95c640 1 Processor -- start 2024-03-23T14:36:29.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.062+0000 7f3f2a95c640 1 -- start start 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f2a95c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3f240d0380 0x7f3f241f6790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f2a95c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f24145570 0x7f3f240a9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f2a95c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f2415cdf0 0x7f3f240aa3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f2a95c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2413c4d0 con 0x7f3f2415cdf0 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f2a95c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3f2406ccf0 con 0x7f3f24145570 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f2a95c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3f2406cfb0 con 0x7f3f240d0380 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f2415cdf0 0x7f3f240aa3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f2415cdf0 0x7f3f240aa3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51796/0 (socket says 172.21.15.161:51796) 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 -- 172.21.15.161:0/2151991612 learned_addr learned my addr 172.21.15.161:0/2151991612 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f23fff640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3f240d0380 0x7f3f241f6790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f237fe640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f24145570 0x7f3f240a9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 -- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3f240d0380 msgr2=0x7f3f241f6790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3f240d0380 0x7f3f241f6790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 -- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f24145570 msgr2=0x7f3f240a9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f24145570 0x7f3f240a9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 -- 172.21.15.161:0/2151991612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f140094d0 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f28ef9640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f2415cdf0 0x7f3f240aa3f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00efc0 tx=0x7f3f1c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f20ff9640 1 -- 172.21.15.161:0/2151991612 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c014480 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f20ff9640 1 -- 172.21.15.161:0/2151991612 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3f1c00f040 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f2406d290 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f20ff9640 1 -- 172.21.15.161:0/2151991612 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f1c014820 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.063+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f2406d850 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.064+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f2406da70 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.065+0000 7f3f20ff9640 1 -- 172.21.15.161:0/2151991612 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3f1c014980 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.065+0000 7f3f20ff9640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3f1006b810 0x7f3f1006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.065+0000 7f3f20ff9640 1 -- 172.21.15.161:0/2151991612 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f3f1c08ad20 con 0x7f3f2415cdf0 2024-03-23T14:36:29.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.065+0000 7f3f23fff640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3f1006b810 0x7f3f1006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.066+0000 7f3f23fff640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3f1006b810 0x7f3f1006dcd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3f14036000 tx=0x7f3f14008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.069+0000 7f3f20ff9640 1 -- 172.21.15.161:0/2151991612 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3f1c0556f0 con 0x7f3f2415cdf0 2024-03-23T14:36:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.242+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3f2404f4e0 con 0x7f3f2415cdf0 2024-03-23T14:36:29.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.242+0000 7f3f20ff9640 1 -- 172.21.15.161:0/2151991612 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f3f1c0180a0 con 0x7f3f2415cdf0 2024-03-23T14:36:29.239 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:29.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3f1006b810 msgr2=0x7f3f1006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3f1006b810 0x7f3f1006dcd0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3f14036000 tx=0x7f3f14008030 comp rx=0 tx=0).stop 2024-03-23T14:36:29.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f2415cdf0 msgr2=0x7f3f240aa3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f2415cdf0 0x7f3f240aa3f0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f3f1c00efc0 tx=0x7f3f1c00c490 comp rx=0 tx=0).stop 2024-03-23T14:36:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 shutdown_connections 2024-03-23T14:36:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3f1006b810 0x7f3f1006dcd0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3f2415cdf0 0x7f3f240aa3f0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3f24145570 0x7f3f240a9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 --2- 172.21.15.161:0/2151991612 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3f240d0380 0x7f3f241f6790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 >> 172.21.15.161:0/2151991612 conn(0x7f3f2406b260 msgr2=0x7f3f240afc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 shutdown_connections 2024-03-23T14:36:29.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.244+0000 7f3f2a95c640 1 -- 172.21.15.161:0/2151991612 wait complete. 2024-03-23T14:36:29.251 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:36:29.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.358+0000 7f3a72815640 1 Processor -- start 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.358+0000 7f3a72815640 1 -- start start 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.359+0000 7f3a72815640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3a6c161750 0x7f3a6c0d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.359+0000 7f3a72815640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a6c0d1330 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.359+0000 7f3a72815640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a6c162e80 con 0x7f3a6c161b30 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.359+0000 7f3a72815640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3a6c0d0720 con 0x7f3a6c161750 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.359+0000 7f3a6bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3a6c161750 0x7f3a6c0d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.359+0000 7f3a6b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3a6c0d1cb0 0x7f3a6c162060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50086/0 (socket says 172.21.15.161:50086) 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.359+0000 7f3a6b7fe640 1 -- 172.21.15.161:0/952823549 learned_addr learned my addr 172.21.15.161:0/952823549 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.360+0000 7f3a6affd640 1 -- 172.21.15.161:0/952823549 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3939599319 0 0) 0x7f3a6c162e80 con 0x7f3a6c161b30 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.360+0000 7f3a6bfff640 1 -- 172.21.15.161:0/952823549 >> v1:172.21.15.183:6789/0 conn(0x7f3a6c161b30 legacy=0x7f3a6c162770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.360+0000 7f3a6bfff640 1 -- 172.21.15.161:0/952823549 >> v1:172.21.15.161:6789/0 conn(0x7f3a6c0d1cb0 legacy=0x7f3a6c162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.360+0000 7f3a6bfff640 1 -- 172.21.15.161:0/952823549 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a6c0d0a00 con 0x7f3a6c161750 2024-03-23T14:36:29.358 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.360+0000 7f3a6bfff640 1 --2- 172.21.15.161:0/952823549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c161750 0x7f3a6c0d25c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0097a0 tx=0x7f3a5c02b790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e28294d34beab878 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a6affd640 1 -- 172.21.15.161:0/952823549 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c030710 con 0x7f3a6c161750 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a6affd640 1 -- 172.21.15.161:0/952823549 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a5c02e050 con 0x7f3a6c161750 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a6affd640 1 -- 172.21.15.161:0/952823549 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c030a90 con 0x7f3a6c161750 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a72815640 1 -- 172.21.15.161:0/952823549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c161750 msgr2=0x7f3a6c0d25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a72815640 1 --2- 172.21.15.161:0/952823549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c161750 0x7f3a6c0d25c0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0097a0 tx=0x7f3a5c02b790 comp rx=0 tx=0).stop 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a72815640 1 -- 172.21.15.161:0/952823549 shutdown_connections 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a72815640 1 --2- 172.21.15.161:0/952823549 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c161750 0x7f3a6c0d25c0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a72815640 1 -- 172.21.15.161:0/952823549 >> 172.21.15.161:0/952823549 conn(0x7f3a6c09d450 msgr2=0x7f3a6c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a72815640 1 -- 172.21.15.161:0/952823549 shutdown_connections 2024-03-23T14:36:29.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.361+0000 7f3a72815640 1 -- 172.21.15.161:0/952823549 wait complete. 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.362+0000 7f3a72815640 1 Processor -- start 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a72815640 1 -- start start 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a72815640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c0d1cb0 0x7f3a6c1f7630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a72815640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c161750 0x7f3a6c0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a72815640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c161b30 0x7f3a6c15c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a72815640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6c1f8560 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a72815640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3a6c1f4060 con 0x7f3a6c161750 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a72815640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3a6c0cdf00 con 0x7f3a6c161b30 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a6bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c0d1cb0 0x7f3a6c1f7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a6bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c0d1cb0 0x7f3a6c1f7630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51808/0 (socket says 172.21.15.161:51808) 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a6bfff640 1 -- 172.21.15.161:0/865608137 learned_addr learned my addr 172.21.15.161:0/865608137 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a6b7fe640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c161750 0x7f3a6c0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.363+0000 7f3a70db2640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c161b30 0x7f3a6c15c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a6bfff640 1 -- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c161b30 msgr2=0x7f3a6c15c730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a6bfff640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c161b30 0x7f3a6c15c730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a6bfff640 1 -- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c161750 msgr2=0x7f3a6c0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a6bfff640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c161750 0x7f3a6c0ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a6bfff640 1 -- 172.21.15.161:0/865608137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a5c009400 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a6bfff640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c0d1cb0 0x7f3a6c1f7630 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0097a0 tx=0x7f3a5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a68ff9640 1 -- 172.21.15.161:0/865608137 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c032910 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a68ff9640 1 -- 172.21.15.161:0/865608137 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a5c032a70 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a6c1f7cc0 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a68ff9640 1 -- 172.21.15.161:0/865608137 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a5c032e20 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.364+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a6c1f8280 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.365+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a6c072c70 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.365+0000 7f3a68ff9640 1 -- 172.21.15.161:0/865608137 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3a5c039040 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.366+0000 7f3a68ff9640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a5806b840 0x7f3a5806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.366+0000 7f3a68ff9640 1 -- 172.21.15.161:0/865608137 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f3a5c0abff0 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.366+0000 7f3a6b7fe640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a5806b840 0x7f3a5806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.366+0000 7f3a6b7fe640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a5806b840 0x7f3a5806dd00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0cdae0 tx=0x7f3a60007e20 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.370+0000 7f3a68ff9640 1 -- 172.21.15.161:0/865608137 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3a5c076930 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.541+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3a6c0a2030 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.542+0000 7f3a68ff9640 1 -- 172.21.15.161:0/865608137 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) ==== 104+0+11 (secure 0 0 0) 0x7f3a5c037020 con 0x7f3a6c0d1cb0 2024-03-23T14:36:29.538 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a5806b840 msgr2=0x7f3a5806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a5806b840 0x7f3a5806dd00 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3a6c0cdae0 tx=0x7f3a60007e20 comp rx=0 tx=0).stop 2024-03-23T14:36:29.540 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c0d1cb0 msgr2=0x7f3a6c1f7630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c0d1cb0 0x7f3a6c1f7630 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3a5c0097a0 tx=0x7f3a5c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 shutdown_connections 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a5806b840 0x7f3a5806dd00 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a6c161b30 0x7f3a6c15c730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a6c161750 0x7f3a6c0ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 --2- 172.21.15.161:0/865608137 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a6c0d1cb0 0x7f3a6c1f7630 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 >> 172.21.15.161:0/865608137 conn(0x7f3a6c09d450 msgr2=0x7f3a6c06c590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 shutdown_connections 2024-03-23T14:36:29.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.544+0000 7f3a72815640 1 -- 172.21.15.161:0/865608137 wait complete. 2024-03-23T14:36:29.551 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:36:29.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.658+0000 7fc32dd0c640 1 Processor -- start 2024-03-23T14:36:29.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.658+0000 7fc32dd0c640 1 -- start start 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc32dd0c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc3281552f0 0x7fc3280b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc32dd0c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc32806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc32dd0c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc32815f480 con 0x7fc3280b2aa0 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc32dd0c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc3281556d0 con 0x7fc3281552f0 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc32dd0c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc32815e870 con 0x7fc3280d0380 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3277fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc3281552f0 0x7fc3280b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3277fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc3281552f0 0x7fc3280b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51810/0 (socket says 172.21.15.161:51810) 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3277fe640 1 -- 172.21.15.161:0/4260135757 learned_addr learned my addr 172.21.15.161:0/4260135757 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc327fff640 1 --2- 172.21.15.161:0/4260135757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc32806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3267fc640 1 -- 172.21.15.161:0/4260135757 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2023034958 0 0) 0x7fc32815f480 con 0x7fc3280b2aa0 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3267fc640 1 -- 172.21.15.161:0/4260135757 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc310003520 con 0x7fc3280b2aa0 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3277fe640 1 -- 172.21.15.161:0/4260135757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 msgr2=0x7fc32806ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3277fe640 1 --2- 172.21.15.161:0/4260135757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc32806ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3277fe640 1 -- 172.21.15.161:0/4260135757 >> v1:172.21.15.183:6789/0 conn(0x7fc3280b2aa0 legacy=0x7fc3280cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc3277fe640 1 -- 172.21.15.161:0/4260135757 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc32815eb50 con 0x7fc3281552f0 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.659+0000 7fc327fff640 1 --2- 172.21.15.161:0/4260135757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc32806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.660+0000 7fc3277fe640 1 --2- 172.21.15.161:0/4260135757 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc3280b3d10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc31800bb10 tx=0x7fc31802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6a6ae92f7876e38 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.660+0000 7fc3267fc640 1 -- 172.21.15.161:0/4260135757 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318036440 con 0x7fc3281552f0 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.660+0000 7fc3267fc640 1 -- 172.21.15.161:0/4260135757 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3180365a0 con 0x7fc3281552f0 2024-03-23T14:36:29.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.660+0000 7fc3267fc640 1 -- 172.21.15.161:0/4260135757 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc318036930 con 0x7fc3281552f0 2024-03-23T14:36:29.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.661+0000 7fc32dd0c640 1 -- 172.21.15.161:0/4260135757 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 msgr2=0x7fc3280b3d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.661+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/4260135757 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc3280b3d10 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fc31800bb10 tx=0x7fc31802e150 comp rx=0 tx=0).stop 2024-03-23T14:36:29.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.661+0000 7fc32dd0c640 1 -- 172.21.15.161:0/4260135757 shutdown_connections 2024-03-23T14:36:29.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.661+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/4260135757 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc32806ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.661+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/4260135757 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc3280b3d10 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.661+0000 7fc32dd0c640 1 -- 172.21.15.161:0/4260135757 >> 172.21.15.161:0/4260135757 conn(0x7fc32806b260 msgr2=0x7fc32806b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:29.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.661+0000 7fc32dd0c640 1 -- 172.21.15.161:0/4260135757 shutdown_connections 2024-03-23T14:36:29.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.661+0000 7fc32dd0c640 1 -- 172.21.15.161:0/4260135757 wait complete. 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.662+0000 7fc32dd0c640 1 Processor -- start 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.662+0000 7fc32dd0c640 1 -- start start 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc32dd0c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280b2aa0 0x7fc32814b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc32dd0c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc3281ef980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc32dd0c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc32815c600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc32dd0c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc32815dec0 con 0x7fc3281552f0 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc32dd0c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc32814ba50 con 0x7fc3280b2aa0 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc32dd0c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc3281ef260 con 0x7fc3280d0380 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc32815c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc32815c600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51822/0 (socket says 172.21.15.161:51822) 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 -- 172.21.15.161:0/1993910544 learned_addr learned my addr 172.21.15.161:0/1993910544 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc326ffd640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc3281ef980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc3277fe640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280b2aa0 0x7fc32814b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 -- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 msgr2=0x7fc3281ef980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc3281ef980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 -- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280b2aa0 msgr2=0x7fc32814b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280b2aa0 0x7fc32814b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 -- 172.21.15.161:0/1993910544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc320009a70 con 0x7fc3281552f0 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc327fff640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc32815c600 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc32000efc0 tx=0x7fc320009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc30ffff640 1 -- 172.21.15.161:0/1993910544 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320009d70 con 0x7fc3281552f0 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc30ffff640 1 -- 172.21.15.161:0/1993910544 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc32000f040 con 0x7fc3281552f0 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc31800b770 con 0x7fc3281552f0 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc30ffff640 1 -- 172.21.15.161:0/1993910544 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc320003990 con 0x7fc3281552f0 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.663+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc32815d620 con 0x7fc3281552f0 2024-03-23T14:36:29.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.664+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc32804f4e0 con 0x7fc3281552f0 2024-03-23T14:36:29.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.665+0000 7fc30ffff640 1 -- 172.21.15.161:0/1993910544 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc3200144a0 con 0x7fc3281552f0 2024-03-23T14:36:29.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.665+0000 7fc30ffff640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc31406b840 0x7fc31406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.665+0000 7fc30ffff640 1 -- 172.21.15.161:0/1993910544 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fc32008ae80 con 0x7fc3281552f0 2024-03-23T14:36:29.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.666+0000 7fc3277fe640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc31406b840 0x7fc31406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.666+0000 7fc3277fe640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc31406b840 0x7fc31406dd00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc318038000 tx=0x7fc318032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.670+0000 7fc30ffff640 1 -- 172.21.15.161:0/1993910544 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc3200557c0 con 0x7fc3281552f0 2024-03-23T14:36:29.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.840+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc32806d7a0 con 0x7fc3281552f0 2024-03-23T14:36:29.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.840+0000 7fc30ffff640 1 -- 172.21.15.161:0/1993910544 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) ==== 100+0+11 (secure 0 0 0) 0x7fc320058e80 con 0x7fc3281552f0 2024-03-23T14:36:29.836 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:36:29.838 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc31406b840 msgr2=0x7fc31406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc31406b840 0x7fc31406dd00 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fc318038000 tx=0x7fc318032000 comp rx=0 tx=0).stop 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 msgr2=0x7fc32815c600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc32815c600 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fc32000efc0 tx=0x7fc320009f90 comp rx=0 tx=0).stop 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 shutdown_connections 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc31406b840 0x7fc31406dd00 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3281552f0 0x7fc32815c600 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3280d0380 0x7fc3281ef980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 --2- 172.21.15.161:0/1993910544 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3280b2aa0 0x7fc32814b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 >> 172.21.15.161:0/1993910544 conn(0x7fc32806b260 msgr2=0x7fc3281523e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.842+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 shutdown_connections 2024-03-23T14:36:29.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.843+0000 7fc32dd0c640 1 -- 172.21.15.161:0/1993910544 wait complete. 2024-03-23T14:36:29.849 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-23T14:36:29.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.959+0000 7f6671f9a640 1 Processor -- start 2024-03-23T14:36:29.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.959+0000 7f6671f9a640 1 -- start start 2024-03-23T14:36:29.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.959+0000 7f6671f9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.959+0000 7f6671f9a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f666c0d1330 con 0x7f666c0d2cd0 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.959+0000 7f6671f9a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f666c0d0720 con 0x7f666c161750 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f6671f9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f666c0d0a00 con 0x7f666c0d1cb0 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f666c0d2cd0 0x7f666c0ceb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50092/0 (socket says 172.21.15.161:50092) 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666affd640 1 -- 172.21.15.161:0/2800711394 learned_addr learned my addr 172.21.15.161:0/2800711394 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666bfff640 1 --2- 172.21.15.161:0/2800711394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666a7fc640 1 -- 172.21.15.161:0/2800711394 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3785766165 0 0) 0x7f666c0d1330 con 0x7f666c0d2cd0 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666a7fc640 1 -- 172.21.15.161:0/2800711394 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6654003520 con 0x7f666c0d2cd0 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666a7fc640 1 -- 172.21.15.161:0/2800711394 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 81678058 0 0) 0x7f666c0d0720 con 0x7f666c161750 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666a7fc640 1 -- 172.21.15.161:0/2800711394 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f666c0d1330 con 0x7f666c161750 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666bfff640 1 -- 172.21.15.161:0/2800711394 >> v1:172.21.15.183:6789/0 conn(0x7f666c161750 legacy=0x7f666c0d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666bfff640 1 -- 172.21.15.161:0/2800711394 >> v1:172.21.15.161:6789/0 conn(0x7f666c0d2cd0 legacy=0x7f666c0ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.960+0000 7f666bfff640 1 -- 172.21.15.161:0/2800711394 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666c0d0ce0 con 0x7f666c0d1cb0 2024-03-23T14:36:29.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f666bfff640 1 --2- 172.21.15.161:0/2800711394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0cf220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6664009dc0 tx=0x7f666402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f654ea2828623805 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f666a7fc640 1 -- 172.21.15.161:0/2800711394 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666402ccd0 con 0x7f666c0d1cb0 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f666a7fc640 1 -- 172.21.15.161:0/2800711394 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666402ce30 con 0x7f666c0d1cb0 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f666a7fc640 1 -- 172.21.15.161:0/2800711394 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664034640 con 0x7f666c0d1cb0 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f6671f9a640 1 -- 172.21.15.161:0/2800711394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 msgr2=0x7f666c0cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f6671f9a640 1 --2- 172.21.15.161:0/2800711394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0cf220 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f6664009dc0 tx=0x7f666402bf20 comp rx=0 tx=0).stop 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f6671f9a640 1 -- 172.21.15.161:0/2800711394 shutdown_connections 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f6671f9a640 1 --2- 172.21.15.161:0/2800711394 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0cf220 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.961+0000 7f6671f9a640 1 -- 172.21.15.161:0/2800711394 >> 172.21.15.161:0/2800711394 conn(0x7f666c09d450 msgr2=0x7f666c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.962+0000 7f6671f9a640 1 -- 172.21.15.161:0/2800711394 shutdown_connections 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.962+0000 7f6671f9a640 1 -- 172.21.15.161:0/2800711394 wait complete. 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.963+0000 7f6671f9a640 1 Processor -- start 2024-03-23T14:36:29.960 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.963+0000 7f6671f9a640 1 -- start start 2024-03-23T14:36:29.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.963+0000 7f6671f9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0b9c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.963+0000 7f6671f9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f666c0d2cd0 0x7f666c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6671f9a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f666c161750 0x7f666c0cd980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6671f9a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f666c0b7520 con 0x7f666c161750 2024-03-23T14:36:29.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6671f9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f666c0ca1b0 con 0x7f666c0d2cd0 2024-03-23T14:36:29.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6671f9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f666c0c9810 con 0x7f666c0d1cb0 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f666c161750 0x7f666c0cd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f666c161750 0x7f666c0cd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51834/0 (socket says 172.21.15.161:51834) 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 -- 172.21.15.161:0/2087042054 learned_addr learned my addr 172.21.15.161:0/2087042054 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666affd640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f666c0d2cd0 0x7f666c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666b7fe640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0b9c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 -- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 msgr2=0x7f666c0b9c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0b9c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 -- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f666c0d2cd0 msgr2=0x7f666c0b6cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f666c0d2cd0 0x7f666c0b6cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 -- 172.21.15.161:0/2087042054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6664009a70 con 0x7f666c161750 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f666bfff640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f666c161750 0x7f666c0cd980 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6664009ee0 tx=0x7f6664004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6653fff640 1 -- 172.21.15.161:0/2087042054 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6664007000 con 0x7f666c161750 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6653fff640 1 -- 172.21.15.161:0/2087042054 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f666403d620 con 0x7f666c161750 2024-03-23T14:36:29.964 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6653fff640 1 -- 172.21.15.161:0/2087042054 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666403d9d0 con 0x7f666c161750 2024-03-23T14:36:29.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666c0c9af0 con 0x7f666c161750 2024-03-23T14:36:29.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.964+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f666c0b7210 con 0x7f666c161750 2024-03-23T14:36:29.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.965+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f666c0ca2f0 con 0x7f666c161750 2024-03-23T14:36:29.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.966+0000 7f6653fff640 1 -- 172.21.15.161:0/2087042054 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f666403b040 con 0x7f666c161750 2024-03-23T14:36:29.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.966+0000 7f6653fff640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b810 0x7f665806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:29.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.966+0000 7f6653fff640 1 -- 172.21.15.161:0/2087042054 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f66640ac760 con 0x7f666c161750 2024-03-23T14:36:29.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.966+0000 7f666b7fe640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b810 0x7f665806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:29.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.967+0000 7f666b7fe640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b810 0x7f665806dcd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f665c007fd0 tx=0x7f665c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:29.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:29.971+0000 7f6653fff640 1 -- 172.21.15.161:0/2087042054 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f666c0ca2f0 con 0x7f666c161750 2024-03-23T14:36:30.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.144+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f666c0ca2f0 con 0x7f666c161750 2024-03-23T14:36:30.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.144+0000 7f6653fff640 1 -- 172.21.15.161:0/2087042054 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v36) ==== 97+0+11 (secure 0 0 0) 0x7f666c0ca2f0 con 0x7f666c161750 2024-03-23T14:36:30.140 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b810 msgr2=0x7f665806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b810 0x7f665806dcd0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f665c007fd0 tx=0x7f665c009040 comp rx=0 tx=0).stop 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f666c161750 msgr2=0x7f666c0cd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f666c161750 0x7f666c0cd980 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f6664009ee0 tx=0x7f6664004a40 comp rx=0 tx=0).stop 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 shutdown_connections 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b810 0x7f665806dcd0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f666c161750 0x7f666c0cd980 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f666c0d2cd0 0x7f666c0b6cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 --2- 172.21.15.161:0/2087042054 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f666c0d1cb0 0x7f666c0b9c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 >> 172.21.15.161:0/2087042054 conn(0x7f666c09d450 msgr2=0x7f666c06bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:30.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 shutdown_connections 2024-03-23T14:36:30.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.146+0000 7f6671f9a640 1 -- 172.21.15.161:0/2087042054 wait complete. 2024-03-23T14:36:30.153 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:36:30.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.259+0000 7f648a1c7640 1 Processor -- start 2024-03-23T14:36:30.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.259+0000 7f648a1c7640 1 -- start start 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f648a1c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f6484072100 0x7f648415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f648a1c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648415be70 con 0x7f64840724e0 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f648a1c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f648415c150 con 0x7f648406eca0 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f648a1c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f648415c430 con 0x7f6484072100 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f6482ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f64840724e0 0x7f648406e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:50094/0 (socket says 172.21.15.161:50094) 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f6482ffd640 1 -- 172.21.15.161:0/885790460 learned_addr learned my addr 172.21.15.161:0/885790460 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f64837fe640 1 --2- 172.21.15.161:0/885790460 >> v2:172.21.15.183:3300/0 conn(0x7f6484072100 0x7f648415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f64827fc640 1 -- 172.21.15.161:0/885790460 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 849484992 0 0) 0x7f648415be70 con 0x7f64840724e0 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f64827fc640 1 -- 172.21.15.161:0/885790460 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f646c003520 con 0x7f64840724e0 2024-03-23T14:36:30.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f64827fc640 1 -- 172.21.15.161:0/885790460 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909942525 0 0) 0x7f648415c150 con 0x7f648406eca0 2024-03-23T14:36:30.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f64827fc640 1 -- 172.21.15.161:0/885790460 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f648415be70 con 0x7f648406eca0 2024-03-23T14:36:30.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f64837fe640 1 -- 172.21.15.161:0/885790460 >> v1:172.21.15.183:6789/0 conn(0x7f648406eca0 legacy=0x7f64840b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f64837fe640 1 -- 172.21.15.161:0/885790460 >> v1:172.21.15.161:6789/0 conn(0x7f64840724e0 legacy=0x7f648406e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.260+0000 7f64837fe640 1 -- 172.21.15.161:0/885790460 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64840b2c70 con 0x7f6484072100 2024-03-23T14:36:30.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.261+0000 7f64837fe640 1 --2- 172.21.15.161:0/885790460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 0x7f648415b3f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f64740097a0 tx=0x7f647402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ad0bb61503ad4e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:30.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.261+0000 7f64827fc640 1 -- 172.21.15.161:0/885790460 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474031830 con 0x7f6484072100 2024-03-23T14:36:30.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.261+0000 7f64827fc640 1 -- 172.21.15.161:0/885790460 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6474031990 con 0x7f6484072100 2024-03-23T14:36:30.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.261+0000 7f64827fc640 1 -- 172.21.15.161:0/885790460 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6474031d00 con 0x7f6484072100 2024-03-23T14:36:30.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.262+0000 7f648a1c7640 1 -- 172.21.15.161:0/885790460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 msgr2=0x7f648415b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.262+0000 7f648a1c7640 1 --2- 172.21.15.161:0/885790460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 0x7f648415b3f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f64740097a0 tx=0x7f647402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.262+0000 7f648a1c7640 1 -- 172.21.15.161:0/885790460 shutdown_connections 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.262+0000 7f648a1c7640 1 --2- 172.21.15.161:0/885790460 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 0x7f648415b3f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.262+0000 7f648a1c7640 1 -- 172.21.15.161:0/885790460 >> 172.21.15.161:0/885790460 conn(0x7f64840b17f0 msgr2=0x7f64840b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.262+0000 7f648a1c7640 1 -- 172.21.15.161:0/885790460 shutdown_connections 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.262+0000 7f648a1c7640 1 -- 172.21.15.161:0/885790460 wait complete. 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.263+0000 7f648a1c7640 1 Processor -- start 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.263+0000 7f648a1c7640 1 -- start start 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f648a1c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f648406eca0 0x7f6484141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f648a1c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 0x7f6484151c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f648a1c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64840724e0 0x7f6484153b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f648a1c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6484150c50 con 0x7f648406eca0 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f648a1c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f64841517d0 con 0x7f6484072100 2024-03-23T14:36:30.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f648a1c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f6484150ea0 con 0x7f64840724e0 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f648406eca0 0x7f6484141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f648406eca0 0x7f6484141620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:51850/0 (socket says 172.21.15.161:51850) 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 -- 172.21.15.161:0/2892901382 learned_addr learned my addr 172.21.15.161:0/2892901382 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f6483fff640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64840724e0 0x7f6484153b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f6482ffd640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 0x7f6484151c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 -- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64840724e0 msgr2=0x7f6484153b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64840724e0 0x7f6484153b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 -- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 msgr2=0x7f6484151c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 0x7f6484151c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 -- 172.21.15.161:0/2892901382 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6474009400 con 0x7f648406eca0 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.264+0000 7f64837fe640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f648406eca0 0x7f6484141620 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6474000c00 tx=0x7f64740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.265+0000 7f646bfff640 1 -- 172.21.15.161:0/2892901382 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647403a830 con 0x7f648406eca0 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.265+0000 7f646bfff640 1 -- 172.21.15.161:0/2892901382 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f647403a9b0 con 0x7f648406eca0 2024-03-23T14:36:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.265+0000 7f646bfff640 1 -- 172.21.15.161:0/2892901382 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f647403ad60 con 0x7f648406eca0 2024-03-23T14:36:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.265+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6484151180 con 0x7f648406eca0 2024-03-23T14:36:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.265+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64841521a0 con 0x7f648406eca0 2024-03-23T14:36:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.266+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f648415c090 con 0x7f648406eca0 2024-03-23T14:36:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.266+0000 7f646bfff640 1 -- 172.21.15.161:0/2892901382 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f647404d440 con 0x7f648406eca0 2024-03-23T14:36:30.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.266+0000 7f646bfff640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f647006b810 0x7f647006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.267+0000 7f646bfff640 1 -- 172.21.15.161:0/2892901382 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f64740ac560 con 0x7f648406eca0 2024-03-23T14:36:30.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.267+0000 7f6482ffd640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f647006b810 0x7f647006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.267+0000 7f6482ffd640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f647006b810 0x7f647006dcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f64840ccab0 tx=0x7f6478009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:30.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.271+0000 7f646bfff640 1 -- 172.21.15.161:0/2892901382 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f6474076d90 con 0x7f648406eca0 2024-03-23T14:36:30.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.454+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f6484152330 con 0x7f648406eca0 2024-03-23T14:36:30.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.454+0000 7f646bfff640 1 -- 172.21.15.161:0/2892901382 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) ==== 80+0+0 (secure 0 0 0) 0x7f647407a450 con 0x7f648406eca0 2024-03-23T14:36:30.450 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:36:30.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f647006b810 msgr2=0x7f647006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f647006b810 0x7f647006dcd0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f64840ccab0 tx=0x7f6478009040 comp rx=0 tx=0).stop 2024-03-23T14:36:30.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f648406eca0 msgr2=0x7f6484141620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f648406eca0 0x7f6484141620 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f6474000c00 tx=0x7f64740046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 shutdown_connections 2024-03-23T14:36:30.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f647006b810 0x7f647006dcd0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64840724e0 0x7f6484153b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6484072100 0x7f6484151c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 --2- 172.21.15.161:0/2892901382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f648406eca0 0x7f6484141620 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 >> 172.21.15.161:0/2892901382 conn(0x7f64840b17f0 msgr2=0x7f648415ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:30.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 shutdown_connections 2024-03-23T14:36:30.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.456+0000 7f648a1c7640 1 -- 172.21.15.161:0/2892901382 wait complete. 2024-03-23T14:36:30.463 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-23T14:36:30.463 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:30.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:30.488 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[52821]: fuse finished with error 0 and tester_r 0 2024-03-23T14:36:30.490 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:36:30.584 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:36:30.584 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:30.584 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T14:36:30.599 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds 'allow rwp' mon 'allow r' osd 'allow rw pool=cephfs_data,allow rw pool=data_new' 2024-03-23T14:36:30.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.751+0000 7f7c5406b640 1 Processor -- start 2024-03-23T14:36:30.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.751+0000 7f7c5406b640 1 -- start start 2024-03-23T14:36:30.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c5406b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7c4c145570 0x7f7c4c0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c5406b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c15cdf0 0x7f7c4c0cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c5406b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c4c0cf200 con 0x7f7c4c0d01b0 2024-03-23T14:36:30.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c5406b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f7c4c0d05f0 con 0x7f7c4c145570 2024-03-23T14:36:30.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c5406b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c15d270 con 0x7f7c4c15cdf0 2024-03-23T14:36:30.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c51e07640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7c4c145570 0x7f7c4c0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c52608640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7c4c0d01b0 0x7f7c4c070b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45090/0 (socket says 172.21.15.161:45090) 2024-03-23T14:36:30.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c51e07640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f7c4c145570 0x7f7c4c0cdf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50564/0 (socket says 172.21.15.161:50564) 2024-03-23T14:36:30.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c52608640 1 -- 172.21.15.161:0/108568225 learned_addr learned my addr 172.21.15.161:0/108568225 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:30.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.752+0000 7f7c51606640 1 --2- 172.21.15.161:0/108568225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c15cdf0 0x7f7c4c0cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.753+0000 7f7c51e07640 1 -- 172.21.15.161:0/108568225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c15cdf0 msgr2=0x7f7c4c0cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:30.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.753+0000 7f7c51e07640 1 --2- 172.21.15.161:0/108568225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c15cdf0 0x7f7c4c0cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.753+0000 7f7c51e07640 1 -- 172.21.15.161:0/108568225 >> v1:172.21.15.161:6789/0 conn(0x7f7c4c0d01b0 legacy=0x7f7c4c070b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.753+0000 7f7c51e07640 1 -- 172.21.15.161:0/108568225 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4c0ce5f0 con 0x7f7c4c145570 2024-03-23T14:36:30.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.753+0000 7f7c50e05640 1 -- 172.21.15.161:0/108568225 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2490481973 0 0) 0x7f7c4c0cf200 con 0x7f7c4c0d01b0 2024-03-23T14:36:30.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.753+0000 7f7c51e07640 1 --2- 172.21.15.161:0/108568225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0cdf30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009d90 tx=0x7f7c4802bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e0ce7e1ea656675 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c50e05640 1 -- 172.21.15.161:0/108568225 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4802cc50 con 0x7f7c4c145570 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c50e05640 1 -- 172.21.15.161:0/108568225 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c4802cdb0 con 0x7f7c4c145570 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c50e05640 1 -- 172.21.15.161:0/108568225 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c48034660 con 0x7f7c4c145570 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c5406b640 1 -- 172.21.15.161:0/108568225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 msgr2=0x7f7c4c0cdf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c5406b640 1 --2- 172.21.15.161:0/108568225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0cdf30 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f7c48009d90 tx=0x7f7c4802bea0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c5406b640 1 -- 172.21.15.161:0/108568225 shutdown_connections 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c5406b640 1 --2- 172.21.15.161:0/108568225 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c15cdf0 0x7f7c4c0cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c5406b640 1 --2- 172.21.15.161:0/108568225 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0cdf30 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c5406b640 1 -- 172.21.15.161:0/108568225 >> 172.21.15.161:0/108568225 conn(0x7f7c4c06b260 msgr2=0x7f7c4c06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:30.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c5406b640 1 -- 172.21.15.161:0/108568225 shutdown_connections 2024-03-23T14:36:30.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.754+0000 7f7c5406b640 1 -- 172.21.15.161:0/108568225 wait complete. 2024-03-23T14:36:30.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.755+0000 7f7c5406b640 1 Processor -- start 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.755+0000 7f7c5406b640 1 -- start start 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c5406b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c0d01b0 0x7f7c4c13f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c5406b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0acb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c5406b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c4c15cdf0 0x7f7c4c1f4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c5406b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c13d890 con 0x7f7c4c145570 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c5406b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4c13d1f0 con 0x7f7c4c15cdf0 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c5406b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7c4c0ad490 con 0x7f7c4c0d01b0 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51606640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0acb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51606640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0acb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50580/0 (socket says 172.21.15.161:50580) 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51606640 1 -- 172.21.15.161:0/512085376 learned_addr learned my addr 172.21.15.161:0/512085376 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c52608640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c4c15cdf0 0x7f7c4c1f4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51e07640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c0d01b0 0x7f7c4c13f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51606640 1 -- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c0d01b0 msgr2=0x7f7c4c13f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51606640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c0d01b0 0x7f7c4c13f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51606640 1 -- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c4c15cdf0 msgr2=0x7f7c4c1f4720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51606640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c4c15cdf0 0x7f7c4c1f4720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.756+0000 7f7c51606640 1 -- 172.21.15.161:0/512085376 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c48009a70 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.757+0000 7f7c51606640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0acb10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00e870 tx=0x7f7c3c00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.757+0000 7f7c427fc640 1 -- 172.21.15.161:0/512085376 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c0094f0 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.757+0000 7f7c427fc640 1 -- 172.21.15.161:0/512085376 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7c3c00c940 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.757+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4c13d4b0 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.757+0000 7f7c427fc640 1 -- 172.21.15.161:0/512085376 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c3c00ccf0 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.757+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4c13ff80 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.758+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c4c13db00 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.758+0000 7f7c427fc640 1 -- 172.21.15.161:0/512085376 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7c3c013470 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.758+0000 7f7c427fc640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7c3006b810 0x7f7c3006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.759+0000 7f7c427fc640 1 -- 172.21.15.161:0/512085376 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f7c3c08a130 con 0x7f7c4c145570 2024-03-23T14:36:30.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.759+0000 7f7c51e07640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7c3006b810 0x7f7c3006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:30.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.759+0000 7f7c51e07640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7c3006b810 0x7f7c3006dcd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7c48036000 tx=0x7f7c48030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:30.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:30.763+0000 7f7c427fc640 1 -- 172.21.15.161:0/512085376 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7c3c054b00 con 0x7f7c4c145570 2024-03-23T14:36:31.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.005+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]} v 0) -- 0x7f7c4c13d050 con 0x7f7c4c145570 2024-03-23T14:36:31.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.007+0000 7f7c427fc640 1 -- 172.21.15.161:0/512085376 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow rwp", "mon", "allow r", "osd", "allow rw pool=cephfs_data,allow rw pool=data_new"]}]=0 updated caps for client.0 v13) ==== 211+0+168 (secure 0 0 0) 0x7f7c4c13d050 con 0x7f7c4c145570 2024-03-23T14:36:31.003 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:36:31.003 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:36:31.003 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow rwp" 2024-03-23T14:36:31.003 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-23T14:36:31.003 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw pool=cephfs_data,allow rw pool=data_new" 2024-03-23T14:36:31.003 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-23T14:36:31.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7c3006b810 msgr2=0x7f7c3006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:31.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7c3006b810 0x7f7c3006dcd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7c48036000 tx=0x7f7c48030000 comp rx=0 tx=0).stop 2024-03-23T14:36:31.005 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 msgr2=0x7f7c4c0acb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0acb10 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f7c3c00e870 tx=0x7f7c3c00ed30 comp rx=0 tx=0).stop 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 shutdown_connections 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7c3006b810 0x7f7c3006dcd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7c4c15cdf0 0x7f7c4c1f4720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7c4c145570 0x7f7c4c0acb10 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 --2- 172.21.15.161:0/512085376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7c4c0d01b0 0x7f7c4c13f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 >> 172.21.15.161:0/512085376 conn(0x7f7c4c06b260 msgr2=0x7f7c4c071480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 shutdown_connections 2024-03-23T14:36:31.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.009+0000 7f7c5406b640 1 -- 172.21.15.161:0/512085376 wait complete. 2024-03-23T14:36:31.016 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:36:31.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.121+0000 7ffae3883640 1 Processor -- start 2024-03-23T14:36:31.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.121+0000 7ffae3883640 1 -- start start 2024-03-23T14:36:31.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae3883640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffadc145080 0x7ffadc072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:31.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae3883640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ffadc1607d0 0x7ffadc15fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:31.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae3883640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffadc0cf4f0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae3883640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ffadc071010 con 0x7ffadc145080 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae3883640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7ffadc0cef30 con 0x7ffadc1607d0 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae161f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffadc145080 0x7ffadc072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae1e20640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ffadc06cbf0 0x7ffadc071430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45096/0 (socket says 172.21.15.161:45096) 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae161f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ffadc145080 0x7ffadc072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50590/0 (socket says 172.21.15.161:50590) 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae1e20640 1 -- 172.21.15.161:0/3367235679 learned_addr learned my addr 172.21.15.161:0/3367235679 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae0e1e640 1 --2- 172.21.15.161:0/3367235679 >> v2:172.21.15.183:3300/0 conn(0x7ffadc1607d0 0x7ffadc15fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae161f640 1 -- 172.21.15.161:0/3367235679 >> v2:172.21.15.183:3300/0 conn(0x7ffadc1607d0 msgr2=0x7ffadc15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae161f640 1 --2- 172.21.15.161:0/3367235679 >> v2:172.21.15.183:3300/0 conn(0x7ffadc1607d0 0x7ffadc15fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae161f640 1 -- 172.21.15.161:0/3367235679 >> v1:172.21.15.161:6789/0 conn(0x7ffadc06cbf0 legacy=0x7ffadc071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffae161f640 1 -- 172.21.15.161:0/3367235679 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffadc0cf210 con 0x7ffadc145080 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.122+0000 7ffad3fff640 1 -- 172.21.15.161:0/3367235679 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2435871286 0 0) 0x7ffadc0cf4f0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.123+0000 7ffae161f640 1 --2- 172.21.15.161:0/3367235679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc145080 0x7ffadc072a00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ffadc073910 tx=0x7ffad802b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1308ca9400349ff5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:31.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.123+0000 7ffad3fff640 1 -- 172.21.15.161:0/3367235679 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad802e070 con 0x7ffadc145080 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.123+0000 7ffad3fff640 1 -- 172.21.15.161:0/3367235679 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffad80315a0 con 0x7ffadc145080 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.123+0000 7ffad3fff640 1 -- 172.21.15.161:0/3367235679 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8031930 con 0x7ffadc145080 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.124+0000 7ffae3883640 1 -- 172.21.15.161:0/3367235679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc145080 msgr2=0x7ffadc072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.124+0000 7ffae3883640 1 --2- 172.21.15.161:0/3367235679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc145080 0x7ffadc072a00 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7ffadc073910 tx=0x7ffad802b840 comp rx=0 tx=0).stop 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.124+0000 7ffae3883640 1 -- 172.21.15.161:0/3367235679 shutdown_connections 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.124+0000 7ffae3883640 1 --2- 172.21.15.161:0/3367235679 >> v2:172.21.15.183:3300/0 conn(0x7ffadc1607d0 0x7ffadc15fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.124+0000 7ffae3883640 1 --2- 172.21.15.161:0/3367235679 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc145080 0x7ffadc072a00 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.124+0000 7ffae3883640 1 -- 172.21.15.161:0/3367235679 >> 172.21.15.161:0/3367235679 conn(0x7ffadc0b17f0 msgr2=0x7ffadc0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.124+0000 7ffae3883640 1 -- 172.21.15.161:0/3367235679 shutdown_connections 2024-03-23T14:36:31.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.124+0000 7ffae3883640 1 -- 172.21.15.161:0/3367235679 wait complete. 2024-03-23T14:36:31.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.125+0000 7ffae3883640 1 Processor -- start 2024-03-23T14:36:31.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.125+0000 7ffae3883640 1 -- start start 2024-03-23T14:36:31.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae3883640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc06cbf0 0x7ffadc0b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:31.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae3883640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffadc145080 0x7ffadc0b3030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:31.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae3883640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffadc1607d0 0x7ffadc13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:31.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae3883640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc13f3f0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae3883640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ffadc0b3730 con 0x7ffadc145080 2024-03-23T14:36:31.125 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae3883640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ffadc0b3a10 con 0x7ffadc1607d0 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc06cbf0 0x7ffadc0b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc06cbf0 0x7ffadc0b47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50598/0 (socket says 172.21.15.161:50598) 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 -- 172.21.15.161:0/3968913192 learned_addr learned my addr 172.21.15.161:0/3968913192 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae0e1e640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffadc145080 0x7ffadc0b3030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae1e20640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffadc1607d0 0x7ffadc13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 -- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffadc1607d0 msgr2=0x7ffadc13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffadc1607d0 0x7ffadc13eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 -- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffadc145080 msgr2=0x7ffadc0b3030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffadc145080 0x7ffadc0b3030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 -- 172.21.15.161:0/3968913192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad8009400 con 0x7ffadc06cbf0 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae161f640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc06cbf0 0x7ffadc0b47d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009890 tx=0x7ffad80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffad1ffb640 1 -- 172.21.15.161:0/3968913192 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8032840 con 0x7ffadc06cbf0 2024-03-23T14:36:31.126 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffad1ffb640 1 -- 172.21.15.161:0/3968913192 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffad80329a0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffadc0b3cf0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.126+0000 7ffad1ffb640 1 -- 172.21.15.161:0/3968913192 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad8032d50 con 0x7ffadc06cbf0 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.127+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffadc0b42b0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.127+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffadc0b1bd0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.128+0000 7ffad1ffb640 1 -- 172.21.15.161:0/3968913192 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ffad80453f0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.128+0000 7ffad1ffb640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffac006b810 0x7ffac006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.128+0000 7ffad1ffb640 1 -- 172.21.15.161:0/3968913192 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7ffad80ac310 con 0x7ffadc06cbf0 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.129+0000 7ffae0e1e640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffac006b810 0x7ffac006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:31.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.129+0000 7ffae0e1e640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffac006b810 0x7ffac006dcd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0af4d0 tx=0x7ffacc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:31.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.133+0000 7ffad1ffb640 1 -- 172.21.15.161:0/3968913192 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ffad8076d70 con 0x7ffadc06cbf0 2024-03-23T14:36:31.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.339+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7ffadc0684f0 con 0x7ffadc06cbf0 2024-03-23T14:36:31.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.339+0000 7ffad1ffb640 1 -- 172.21.15.161:0/3968913192 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+82 (secure 0 0 0) 0x7ffad807a430 con 0x7ffadc06cbf0 2024-03-23T14:36:31.336 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data data_new ] 2024-03-23T14:36:31.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffac006b810 msgr2=0x7ffac006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:31.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffac006b810 0x7ffac006dcd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7ffadc0af4d0 tx=0x7ffacc009190 comp rx=0 tx=0).stop 2024-03-23T14:36:31.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc06cbf0 msgr2=0x7ffadc0b47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:31.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc06cbf0 0x7ffadc0b47d0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7ffad8009890 tx=0x7ffad80046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 shutdown_connections 2024-03-23T14:36:31.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ffac006b810 0x7ffac006dcd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.338 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ffadc1607d0 0x7ffadc13eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ffadc145080 0x7ffadc0b3030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 --2- 172.21.15.161:0/3968913192 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ffadc06cbf0 0x7ffadc0b47d0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:31.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 >> 172.21.15.161:0/3968913192 conn(0x7ffadc0b17f0 msgr2=0x7ffadc06d0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:31.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 shutdown_connections 2024-03-23T14:36:31.339 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31.341+0000 7ffae3883640 1 -- 172.21.15.161:0/3968913192 wait complete. 2024-03-23T14:36:31.348 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:36:31.348 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:36:31.348 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-23T14:36:31.348 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T14:36:31.349 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:31.349 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:36:31.349 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:36:31.349 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:36:31.349 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:31.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:31.349 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-23T14:36:31.365 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:36:31.365 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:36:31.365 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:36:31.365 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:31.365 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:36:31.365 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:31.365 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:31.366 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42679sec preferred_lft 42679sec 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c4a9:64ff:fe59:877a/64 scope link 2024-03-23T14:36:31.367 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:31.368 INFO:teuthology.orchestra.run.smithi161.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:36:31.368 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:31.368 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c48:50ff:fe01:9f4e/64 scope link 2024-03-23T14:36:31.368 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:31.368 INFO:teuthology.orchestra.run.smithi161.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:36:31.368 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 62:02:d5:17:e7:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:31.368 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::6002:d5ff:fe17:e70d/64 scope link 2024-03-23T14:36:31.369 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:36:31.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:31.369 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:36:31.431 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-23T14:36:31.432 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:31.433 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-23T14:36:31.488 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-23T14:36:31.488 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:36:31.488 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:36:31.488 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:36:31.488 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:36:31.489 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:36:31.489 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:36:31.489 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:36:31.489 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:36:31.489 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:36:31.489 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:36:31.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:36:31.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:31 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:36:31.624 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:31.624 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:36:31.639 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T14:36:31.639 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T14:36:31.640 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:31.640 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:31.640 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-23T14:36:31.706 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T14:36:31.707 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:36:31.707 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:31.761 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:36:31.762 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:36:31.762 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:31.816 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:36:31.884 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:36:31.940 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:36:31.940 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:36:31.940 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:36:31.940 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:36:31.940 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:36:31.940 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:36:31.940 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:36:31.941 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:31.942 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:31.943 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:31.943 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:36:31.995 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T14:36:31.995 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T14:36:32.038 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:36:32.065 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:36:32.121 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:36:32.122 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:36:32.123 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:36:32.124 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:36:32.124 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:36:32.124 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:36:32.124 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:32.124 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:36:32.124 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:32.124 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:36:32.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:32.124 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:36:32.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:2024-03-23T14:36:32.128+0000 7fa0878b7480 -1 init, newargv = 0x56088f0189e0 newargc=15 2024-03-23T14:36:32.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[53885]: starting ceph client 2024-03-23T14:36:32.128 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[53885]: starting fuse 2024-03-23T14:36:32.139 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:36:32.140 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T14:36:32.140 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:36:32.141 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.0")) 2024-03-23T14:36:32.142 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:36:32.260 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.0.53885.asok 2024-03-23T14:36:32.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:36:32.270 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.53885.asok 2024-03-23T14:36:32.270 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:32.270 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.53885.asok status 2024-03-23T14:36:32.406 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:36:32.406 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-23T14:36:32.406 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:36:32.406 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:36:32.406 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "0", 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "53885", 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5685, 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 5685 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:32.407 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3548741726 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3548741726 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.5685 192.168.0.1:0/3548741726", 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/3548741726", 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-23T14:36:32.408 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 16, 2024-03-23T14:36:32.409 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 36, 2024-03-23T14:36:32.409 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:36:32.409 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-23T14:36:32.409 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-23T14:36:32.409 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:36:32.415 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:32.415 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-23T14:36:32.415 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:32.430 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-23T14:36:32.431 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:32.431 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:32.431 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:32.498 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:32.498 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.file.layout.pool -v data_new layoutfile) 2024-03-23T14:36:32.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:32.555 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest/mnt.0 && exec setfattr -n ceph.dir.layout.pool -v data_new layoutdir) 2024-03-23T14:36:32.615 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:32.615 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:32.671 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:32.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.814+0000 7f221b379640 1 Processor -- start 2024-03-23T14:36:32.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.814+0000 7f221b379640 1 -- start start 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f221b379640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2214145080 0x7f2214072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f221b379640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22141607d0 0x7f221415fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f221b379640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22140cf4f0 con 0x7f221406cbf0 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f221b379640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2214071010 con 0x7f22141607d0 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f221b379640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f22140cef30 con 0x7f2214145080 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f2218914640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22141607d0 0x7f221415fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f2218914640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22141607d0 0x7f221415fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50606/0 (socket says 172.21.15.161:50606) 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f2218914640 1 -- 172.21.15.161:0/3998469382 learned_addr learned my addr 172.21.15.161:0/3998469382 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.815+0000 7f2219115640 1 --2- 172.21.15.161:0/3998469382 >> v2:172.21.15.183:3300/0 conn(0x7f2214145080 0x7f2214072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.816+0000 7f2218914640 1 -- 172.21.15.161:0/3998469382 >> v2:172.21.15.183:3300/0 conn(0x7f2214145080 msgr2=0x7f2214072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.816+0000 7f2218914640 1 --2- 172.21.15.161:0/3998469382 >> v2:172.21.15.183:3300/0 conn(0x7f2214145080 0x7f2214072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.816+0000 7f2218914640 1 -- 172.21.15.161:0/3998469382 >> v1:172.21.15.183:6789/0 conn(0x7f221406cbf0 legacy=0x7f2214071430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:32.812 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.816+0000 7f220bfff640 1 -- 172.21.15.161:0/3998469382 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3313332467 0 0) 0x7f22140cf4f0 con 0x7f221406cbf0 2024-03-23T14:36:32.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.816+0000 7f2218914640 1 -- 172.21.15.161:0/3998469382 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22140cf210 con 0x7f22141607d0 2024-03-23T14:36:32.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.816+0000 7f2218914640 1 --2- 172.21.15.161:0/3998469382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22141607d0 0x7f221415fae0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f221406c9a0 tx=0x7f220402b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb06930e0d2a0cca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:32.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f220bfff640 1 -- 172.21.15.161:0/3998469382 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204031870 con 0x7f22141607d0 2024-03-23T14:36:32.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f220bfff640 1 -- 172.21.15.161:0/3998469382 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22040319d0 con 0x7f22141607d0 2024-03-23T14:36:32.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f220bfff640 1 -- 172.21.15.161:0/3998469382 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204031d60 con 0x7f22141607d0 2024-03-23T14:36:32.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f221b379640 1 -- 172.21.15.161:0/3998469382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22141607d0 msgr2=0x7f221415fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:32.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f221b379640 1 --2- 172.21.15.161:0/3998469382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22141607d0 0x7f221415fae0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f221406c9a0 tx=0x7f220402b910 comp rx=0 tx=0).stop 2024-03-23T14:36:32.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f221b379640 1 -- 172.21.15.161:0/3998469382 shutdown_connections 2024-03-23T14:36:32.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f221b379640 1 --2- 172.21.15.161:0/3998469382 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22141607d0 0x7f221415fae0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f221b379640 1 --2- 172.21.15.161:0/3998469382 >> v2:172.21.15.183:3300/0 conn(0x7f2214145080 0x7f2214072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f221b379640 1 -- 172.21.15.161:0/3998469382 >> 172.21.15.161:0/3998469382 conn(0x7f22140b17f0 msgr2=0x7f22140b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:32.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f221b379640 1 -- 172.21.15.161:0/3998469382 shutdown_connections 2024-03-23T14:36:32.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.817+0000 7f221b379640 1 -- 172.21.15.161:0/3998469382 wait complete. 2024-03-23T14:36:32.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.818+0000 7f221b379640 1 Processor -- start 2024-03-23T14:36:32.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.818+0000 7f221b379640 1 -- start start 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f221b379640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f221406cbf0 0x7f221413bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f221b379640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2214145080 0x7f221413a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f221b379640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22141607d0 0x7f221415e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f221b379640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f221415aae0 con 0x7f221406cbf0 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f221b379640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f221413c2e0 con 0x7f22141607d0 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f221b379640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f22141f5e20 con 0x7f2214145080 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2219115640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f221406cbf0 0x7f221413bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2219115640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f221406cbf0 0x7f221413bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50620/0 (socket says 172.21.15.161:50620) 2024-03-23T14:36:32.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2219115640 1 -- 172.21.15.161:0/4202685910 learned_addr learned my addr 172.21.15.161:0/4202685910 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2219916640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22141607d0 0x7f221415e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2218914640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2214145080 0x7f221413a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2219115640 1 -- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2214145080 msgr2=0x7f221413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2219115640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2214145080 0x7f221413a7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2219115640 1 -- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22141607d0 msgr2=0x7f221415e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.819+0000 7f2219115640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22141607d0 0x7f221415e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.820+0000 7f2219115640 1 -- 172.21.15.161:0/4202685910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22040094d0 con 0x7f221406cbf0 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.820+0000 7f2219115640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f221406cbf0 0x7f221413bda0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f221000e9b0 tx=0x7f221000ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.820+0000 7f2209ffb640 1 -- 172.21.15.161:0/4202685910 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2210009670 con 0x7f221406cbf0 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.820+0000 7f2209ffb640 1 -- 172.21.15.161:0/4202685910 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f221000c9a0 con 0x7f221406cbf0 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.820+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f221415a1b0 con 0x7f221406cbf0 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.820+0000 7f2209ffb640 1 -- 172.21.15.161:0/4202685910 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f221000cd50 con 0x7f221406cbf0 2024-03-23T14:36:32.818 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.820+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f221415a770 con 0x7f221406cbf0 2024-03-23T14:36:32.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.821+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22140735e0 con 0x7f221406cbf0 2024-03-23T14:36:32.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.822+0000 7f2209ffb640 1 -- 172.21.15.161:0/4202685910 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2210013470 con 0x7f221406cbf0 2024-03-23T14:36:32.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.822+0000 7f2209ffb640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21f806b810 0x7f21f806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:32.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.822+0000 7f2209ffb640 1 -- 172.21.15.161:0/4202685910 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f221008a550 con 0x7f221406cbf0 2024-03-23T14:36:32.819 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.822+0000 7f2218914640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21f806b810 0x7f21f806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:32.820 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.822+0000 7f2218914640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21f806b810 0x7f21f806dcd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2204036000 tx=0x7f220403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:32.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.826+0000 7f2209ffb640 1 -- 172.21.15.161:0/4202685910 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2210054e90 con 0x7f221406cbf0 2024-03-23T14:36:32.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.998+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22140ccc00 con 0x7f221406cbf0 2024-03-23T14:36:32.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:32.999+0000 7f2209ffb640 1 -- 172.21.15.161:0/4202685910 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7f2210058550 con 0x7f221406cbf0 2024-03-23T14:36:32.995 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:32.995 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:32.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21f806b810 msgr2=0x7f21f806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:32.997 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21f806b810 0x7f21f806dcd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2204036000 tx=0x7f220403c040 comp rx=0 tx=0).stop 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f221406cbf0 msgr2=0x7f221413bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f221406cbf0 0x7f221413bda0 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f221000e9b0 tx=0x7f221000ee70 comp rx=0 tx=0).stop 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 shutdown_connections 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f21f806b810 0x7f21f806dcd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22141607d0 0x7f221415e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2214145080 0x7f221413a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 --2- 172.21.15.161:0/4202685910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f221406cbf0 0x7f221413bda0 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.001+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 >> 172.21.15.161:0/4202685910 conn(0x7f22140b17f0 msgr2=0x7f22140b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.002+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 shutdown_connections 2024-03-23T14:36:32.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.002+0000 7f221b379640 1 -- 172.21.15.161:0/4202685910 wait complete. 2024-03-23T14:36:33.008 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:04.688180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:05.241774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-23T14:36:18.738957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:33.009 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:33.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.117+0000 7f202e42e640 1 Processor -- start 2024-03-23T14:36:33.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.117+0000 7f202e42e640 1 -- start start 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.117+0000 7f202e42e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f202815dfe0 0x7f20280d2b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.117+0000 7f202e42e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f202815e3c0 0x7f20280d4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.117+0000 7f202e42e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 0x7f20280d0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.117+0000 7f202e42e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f20280d1f60 con 0x7f202815e3c0 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.117+0000 7f202e42e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f20280d1a30 con 0x7f202815dfe0 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.117+0000 7f202e42e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f20280ce9e0 con 0x7f20280cf760 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f20277fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f202815e3c0 0x7f20280d4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f20277fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f202815e3c0 0x7f20280d4980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50630/0 (socket says 172.21.15.161:50630) 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f20277fe640 1 -- 172.21.15.161:0/857987813 learned_addr learned my addr 172.21.15.161:0/857987813 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f2027fff640 1 --2- 172.21.15.161:0/857987813 >> v2:172.21.15.183:3300/0 conn(0x7f202815dfe0 0x7f20280d2b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f202c9cb640 1 --2- 172.21.15.161:0/857987813 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 0x7f20280d0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f20277fe640 1 -- 172.21.15.161:0/857987813 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 msgr2=0x7f20280d0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:33.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f20277fe640 1 --2- 172.21.15.161:0/857987813 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 0x7f20280d0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f20277fe640 1 -- 172.21.15.161:0/857987813 >> v2:172.21.15.183:3300/0 conn(0x7f202815dfe0 msgr2=0x7f20280d2b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:33.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f20277fe640 1 --2- 172.21.15.161:0/857987813 >> v2:172.21.15.183:3300/0 conn(0x7f202815dfe0 0x7f20280d2b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.118+0000 7f20277fe640 1 -- 172.21.15.161:0/857987813 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20280d1c80 con 0x7f202815e3c0 2024-03-23T14:36:33.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f20277fe640 1 --2- 172.21.15.161:0/857987813 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815e3c0 0x7f20280d4980 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f20280b6a50 tx=0x7f201c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41cbf5fe63d75f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f2026ffd640 1 -- 172.21.15.161:0/857987813 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c031870 con 0x7f202815e3c0 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f2026ffd640 1 -- 172.21.15.161:0/857987813 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f201c0319d0 con 0x7f202815e3c0 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f2026ffd640 1 -- 172.21.15.161:0/857987813 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c031d60 con 0x7f202815e3c0 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 -- 172.21.15.161:0/857987813 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815e3c0 msgr2=0x7f20280d4980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 --2- 172.21.15.161:0/857987813 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815e3c0 0x7f20280d4980 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f20280b6a50 tx=0x7f201c02b910 comp rx=0 tx=0).stop 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 -- 172.21.15.161:0/857987813 shutdown_connections 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 --2- 172.21.15.161:0/857987813 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 0x7f20280d0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 --2- 172.21.15.161:0/857987813 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815e3c0 0x7f20280d4980 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 --2- 172.21.15.161:0/857987813 >> v2:172.21.15.183:3300/0 conn(0x7f202815dfe0 0x7f20280d2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 -- 172.21.15.161:0/857987813 >> 172.21.15.161:0/857987813 conn(0x7f20280d3e60 msgr2=0x7f20280d4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 -- 172.21.15.161:0/857987813 shutdown_connections 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.119+0000 7f202e42e640 1 -- 172.21.15.161:0/857987813 wait complete. 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.121+0000 7f202e42e640 1 Processor -- start 2024-03-23T14:36:33.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.121+0000 7f202e42e640 1 -- start start 2024-03-23T14:36:33.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.121+0000 7f202e42e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 0x7f20281f4730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.121+0000 7f202e42e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815dfe0 0x7f20280ca370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.121+0000 7f202e42e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f202815e3c0 0x7f20280ccbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.121+0000 7f202e42e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f20280cc070 con 0x7f202815dfe0 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.121+0000 7f202e42e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f20280cb4f0 con 0x7f202815e3c0 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.121+0000 7f202e42e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f20280cb7d0 con 0x7f20280cf760 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815dfe0 0x7f20280ca370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815dfe0 0x7f20280ca370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50642/0 (socket says 172.21.15.161:50642) 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 -- 172.21.15.161:0/650263223 learned_addr learned my addr 172.21.15.161:0/650263223 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f2027fff640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 0x7f20281f4730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f202c9cb640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f202815e3c0 0x7f20280ccbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 -- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 msgr2=0x7f20281f4730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 0x7f20281f4730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 -- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f202815e3c0 msgr2=0x7f20280ccbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f202815e3c0 0x7f20280ccbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 -- 172.21.15.161:0/650263223 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f201c0094d0 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f20277fe640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815dfe0 0x7f20280ca370 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f201c02b950 tx=0x7f201c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f2024ff9640 1 -- 172.21.15.161:0/650263223 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c03abe0 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f2024ff9640 1 -- 172.21.15.161:0/650263223 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f201c03ad60 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f2024ff9640 1 -- 172.21.15.161:0/650263223 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f201c044630 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20280cbab0 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.122+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20280ca8b0 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.123+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20280b94b0 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.124+0000 7f2024ff9640 1 -- 172.21.15.161:0/650263223 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f201c04d450 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.124+0000 7f2024ff9640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f201406b810 0x7f201406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.124+0000 7f2024ff9640 1 -- 172.21.15.161:0/650263223 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f201c02d040 con 0x7f202815dfe0 2024-03-23T14:36:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.124+0000 7f2027fff640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f201406b810 0x7f201406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.125+0000 7f2027fff640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f201406b810 0x7f201406dcd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2018009b40 tx=0x7f2018009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.128+0000 7f2024ff9640 1 -- 172.21.15.161:0/650263223 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f201c077120 con 0x7f202815dfe0 2024-03-23T14:36:33.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.300+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f202804f4e0 con 0x7f202815dfe0 2024-03-23T14:36:33.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.300+0000 7f2024ff9640 1 -- 172.21.15.161:0/650263223 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f201c07a7e0 con 0x7f202815dfe0 2024-03-23T14:36:33.296 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:33.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f201406b810 msgr2=0x7f201406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f201406b810 0x7f201406dcd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f2018009b40 tx=0x7f2018009190 comp rx=0 tx=0).stop 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815dfe0 msgr2=0x7f20280ca370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815dfe0 0x7f20280ca370 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f201c02b950 tx=0x7f201c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 shutdown_connections 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f201406b810 0x7f201406dcd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f202815e3c0 0x7f20280ccbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f202815dfe0 0x7f20280ca370 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 --2- 172.21.15.161:0/650263223 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20280cf760 0x7f20281f4730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 >> 172.21.15.161:0/650263223 conn(0x7f20280d3e60 msgr2=0x7f202806cc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 shutdown_connections 2024-03-23T14:36:33.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.302+0000 7f202e42e640 1 -- 172.21.15.161:0/650263223 wait complete. 2024-03-23T14:36:33.309 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:36:33.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.417+0000 7f05517c9640 1 Processor -- start 2024-03-23T14:36:33.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.417+0000 7f05517c9640 1 -- start start 2024-03-23T14:36:33.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f05517c9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f054c15be50 0x7f054c0d0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f05517c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0af320 0x7f054c0d0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f05517c9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f054c0b2f30 con 0x7f054c0cc740 2024-03-23T14:36:33.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f05517c9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f054c0b0400 con 0x7f054c15be50 2024-03-23T14:36:33.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f05517c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f054c0af7f0 con 0x7f054c0af320 2024-03-23T14:36:33.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f054affd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f054c15be50 0x7f054c0d0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f054a7fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f054c0cc740 0x7f054c0aec10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:59524/0 (socket says 172.21.15.161:59524) 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f054a7fc640 1 -- 172.21.15.161:0/921162950 learned_addr learned my addr 172.21.15.161:0/921162950 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f054b7fe640 1 --2- 172.21.15.161:0/921162950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0af320 0x7f054c0d0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f054affd640 1 -- 172.21.15.161:0/921162950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0af320 msgr2=0x7f054c0d0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f054affd640 1 --2- 172.21.15.161:0/921162950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0af320 0x7f054c0d0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f054affd640 1 -- 172.21.15.161:0/921162950 >> v1:172.21.15.183:6789/0 conn(0x7f054c0cc740 legacy=0x7f054c0aec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.418+0000 7f054affd640 1 -- 172.21.15.161:0/921162950 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c0afad0 con 0x7f054c15be50 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.419+0000 7f054affd640 1 --2- 172.21.15.161:0/921162950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 0x7f054c0d0120 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f053c0097a0 tx=0x7f053c02b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a9c6bdd3c69e37a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.419+0000 7f0549ffb640 1 -- 172.21.15.161:0/921162950 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c031830 con 0x7f054c15be50 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.419+0000 7f0549ffb640 1 -- 172.21.15.161:0/921162950 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053c031990 con 0x7f054c15be50 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.419+0000 7f0549ffb640 1 -- 172.21.15.161:0/921162950 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c031d20 con 0x7f054c15be50 2024-03-23T14:36:33.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.420+0000 7f05517c9640 1 -- 172.21.15.161:0/921162950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 msgr2=0x7f054c0d0120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.420+0000 7f05517c9640 1 --2- 172.21.15.161:0/921162950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 0x7f054c0d0120 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f053c0097a0 tx=0x7f053c02b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.420+0000 7f05517c9640 1 -- 172.21.15.161:0/921162950 shutdown_connections 2024-03-23T14:36:33.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.420+0000 7f05517c9640 1 --2- 172.21.15.161:0/921162950 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0af320 0x7f054c0d0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.420+0000 7f05517c9640 1 --2- 172.21.15.161:0/921162950 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 0x7f054c0d0120 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.420+0000 7f05517c9640 1 -- 172.21.15.161:0/921162950 >> 172.21.15.161:0/921162950 conn(0x7f054c09aa40 msgr2=0x7f054c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:33.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.420+0000 7f05517c9640 1 -- 172.21.15.161:0/921162950 shutdown_connections 2024-03-23T14:36:33.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.420+0000 7f05517c9640 1 -- 172.21.15.161:0/921162950 wait complete. 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.421+0000 7f05517c9640 1 Processor -- start 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.421+0000 7f05517c9640 1 -- start start 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f05517c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f054c0af320 0x7f054c148d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f05517c9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0cc740 0x7f054c0c8e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f05517c9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 0x7f054c0c9370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f05517c9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f054c0b7cf0 con 0x7f054c0af320 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f05517c9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f054c149260 con 0x7f054c15be50 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f05517c9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f054c15e4c0 con 0x7f054c0cc740 2024-03-23T14:36:33.421 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f054c0af320 0x7f054c148d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f054c0af320 0x7f054c148d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50646/0 (socket says 172.21.15.161:50646) 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054affd640 1 -- 172.21.15.161:0/2352021493 learned_addr learned my addr 172.21.15.161:0/2352021493 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054b7fe640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 0x7f054c0c9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054a7fc640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0cc740 0x7f054c0c8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054affd640 1 -- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0cc740 msgr2=0x7f054c0c8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054affd640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0cc740 0x7f054c0c8e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054affd640 1 -- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 msgr2=0x7f054c0c9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054affd640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 0x7f054c0c9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.422+0000 7f054affd640 1 -- 172.21.15.161:0/2352021493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f053c009400 con 0x7f054c0af320 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.423+0000 7f054affd640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f054c0af320 0x7f054c148d20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f053c0098c0 tx=0x7f053c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.423+0000 7f05337fe640 1 -- 172.21.15.161:0/2352021493 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c03a8f0 con 0x7f054c0af320 2024-03-23T14:36:33.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.423+0000 7f05337fe640 1 -- 172.21.15.161:0/2352021493 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f053c03aa70 con 0x7f054c0af320 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.423+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c15e7a0 con 0x7f054c0af320 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.423+0000 7f05337fe640 1 -- 172.21.15.161:0/2352021493 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f053c03ae20 con 0x7f054c0af320 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.423+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c15ed60 con 0x7f054c0af320 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.424+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f054c09ff90 con 0x7f054c0af320 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.424+0000 7f05337fe640 1 -- 172.21.15.161:0/2352021493 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f053c04d440 con 0x7f054c0af320 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.424+0000 7f05337fe640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f053806b810 0x7f053806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.425+0000 7f05337fe640 1 -- 172.21.15.161:0/2352021493 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f053c0ac2d0 con 0x7f054c0af320 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.425+0000 7f054a7fc640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f053806b810 0x7f053806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.425+0000 7f054a7fc640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f053806b810 0x7f053806dcd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f054c0b2c90 tx=0x7f0540007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.429+0000 7f05337fe640 1 -- 172.21.15.161:0/2352021493 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f053c076c20 con 0x7f054c0af320 2024-03-23T14:36:33.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.601+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f054c04f550 con 0x7f054c0af320 2024-03-23T14:36:33.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.602+0000 7f05337fe640 1 -- 172.21.15.161:0/2352021493 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) ==== 104+0+11 (secure 0 0 0) 0x7f053c07a2e0 con 0x7f054c0af320 2024-03-23T14:36:33.598 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f053806b810 msgr2=0x7f053806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f053806b810 0x7f053806dcd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f054c0b2c90 tx=0x7f0540007e80 comp rx=0 tx=0).stop 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f054c0af320 msgr2=0x7f054c148d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f054c0af320 0x7f054c148d20 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f053c0098c0 tx=0x7f053c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 shutdown_connections 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f053806b810 0x7f053806dcd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f054c15be50 0x7f054c0c9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f054c0cc740 0x7f054c0c8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 --2- 172.21.15.161:0/2352021493 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f054c0af320 0x7f054c148d20 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 >> 172.21.15.161:0/2352021493 conn(0x7f054c09aa40 msgr2=0x7f054c070240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:33.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 shutdown_connections 2024-03-23T14:36:33.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.604+0000 7f05517c9640 1 -- 172.21.15.161:0/2352021493 wait complete. 2024-03-23T14:36:33.611 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:36:33.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.718+0000 7f182193e640 1 Processor -- start 2024-03-23T14:36:33.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.718+0000 7f182193e640 1 -- start start 2024-03-23T14:36:33.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f182193e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f181c0b2aa0 0x7f181c0cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f182193e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0d01b0 0x7f181c06ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f182193e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f181c0d0600 con 0x7f181c1552f0 2024-03-23T14:36:33.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f182193e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f181c15f480 con 0x7f181c0b2aa0 2024-03-23T14:36:33.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f182193e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f181c1556d0 con 0x7f181c0d01b0 2024-03-23T14:36:33.716 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f181c1552f0 0x7f181c0b3d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45110/0 (socket says 172.21.15.161:45110) 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181affd640 1 -- 172.21.15.161:0/1212599632 learned_addr learned my addr 172.21.15.161:0/1212599632 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181b7fe640 1 --2- 172.21.15.161:0/1212599632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0d01b0 0x7f181c06ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f18127fc640 1 --2- 172.21.15.161:0/1212599632 >> v2:172.21.15.183:3300/0 conn(0x7f181c0b2aa0 0x7f181c0cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181a7fc640 1 -- 172.21.15.161:0/1212599632 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3351289023 0 0) 0x7f180c001110 con 0x7f181c1552f0 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181a7fc640 1 -- 172.21.15.161:0/1212599632 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1808003520 con 0x7f181c1552f0 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181b7fe640 1 -- 172.21.15.161:0/1212599632 >> v2:172.21.15.183:3300/0 conn(0x7f181c0b2aa0 msgr2=0x7f181c0cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181b7fe640 1 --2- 172.21.15.161:0/1212599632 >> v2:172.21.15.183:3300/0 conn(0x7f181c0b2aa0 0x7f181c0cfc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181b7fe640 1 -- 172.21.15.161:0/1212599632 >> v1:172.21.15.161:6789/0 conn(0x7f181c1552f0 legacy=0x7f181c0b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.719+0000 7f181b7fe640 1 -- 172.21.15.161:0/1212599632 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f181c15e870 con 0x7f181c0d01b0 2024-03-23T14:36:33.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.720+0000 7f181b7fe640 1 --2- 172.21.15.161:0/1212599632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0d01b0 0x7f181c06ded0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1814009d90 tx=0x7f181402bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d22e0095cb9c5571 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.720+0000 7f181a7fc640 1 -- 172.21.15.161:0/1212599632 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f181402cc50 con 0x7f181c0d01b0 2024-03-23T14:36:33.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.720+0000 7f181a7fc640 1 -- 172.21.15.161:0/1212599632 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f181402cdb0 con 0x7f181c0d01b0 2024-03-23T14:36:33.718 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.720+0000 7f181a7fc640 1 -- 172.21.15.161:0/1212599632 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1814034640 con 0x7f181c0d01b0 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.721+0000 7f182193e640 1 -- 172.21.15.161:0/1212599632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0d01b0 msgr2=0x7f181c06ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.721+0000 7f182193e640 1 --2- 172.21.15.161:0/1212599632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0d01b0 0x7f181c06ded0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1814009d90 tx=0x7f181402bea0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.721+0000 7f182193e640 1 -- 172.21.15.161:0/1212599632 shutdown_connections 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.721+0000 7f182193e640 1 --2- 172.21.15.161:0/1212599632 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0d01b0 0x7f181c06ded0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.721+0000 7f182193e640 1 --2- 172.21.15.161:0/1212599632 >> v2:172.21.15.183:3300/0 conn(0x7f181c0b2aa0 0x7f181c0cfc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.721+0000 7f182193e640 1 -- 172.21.15.161:0/1212599632 >> 172.21.15.161:0/1212599632 conn(0x7f181c06b260 msgr2=0x7f181c06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.721+0000 7f182193e640 1 -- 172.21.15.161:0/1212599632 shutdown_connections 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.721+0000 7f182193e640 1 -- 172.21.15.161:0/1212599632 wait complete. 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.722+0000 7f182193e640 1 Processor -- start 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.722+0000 7f182193e640 1 -- start start 2024-03-23T14:36:33.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f182193e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0b2aa0 0x7f181c14e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f182193e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f181c0d01b0 0x7f181c1558a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f182193e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f181c1552f0 0x7f181c155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f182193e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f181c157820 con 0x7f181c0d01b0 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f182193e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f181c13ff10 con 0x7f181c1552f0 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f182193e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f181c14ec30 con 0x7f181c0b2aa0 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f18127fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f181c0d01b0 0x7f181c1558a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f18127fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f181c0d01b0 0x7f181c1558a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50648/0 (socket says 172.21.15.161:50648) 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f18127fc640 1 -- 172.21.15.161:0/2260149689 learned_addr learned my addr 172.21.15.161:0/2260149689 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f181affd640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0b2aa0 0x7f181c14e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f181b7fe640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f181c1552f0 0x7f181c155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f18127fc640 1 -- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0b2aa0 msgr2=0x7f181c14e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f18127fc640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0b2aa0 0x7f181c14e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f18127fc640 1 -- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f181c1552f0 msgr2=0x7f181c155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f18127fc640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f181c1552f0 0x7f181c155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.723+0000 7f18127fc640 1 -- 172.21.15.161:0/2260149689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1814009a70 con 0x7f181c0d01b0 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.724+0000 7f18127fc640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f181c0d01b0 0x7f181c1558a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f180400e870 tx=0x7f180400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.724+0000 7f1818ff9640 1 -- 172.21.15.161:0/2260149689 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18040094f0 con 0x7f181c0d01b0 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.724+0000 7f1818ff9640 1 -- 172.21.15.161:0/2260149689 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f180400c940 con 0x7f181c0d01b0 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.724+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f181c140120 con 0x7f181c0d01b0 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.724+0000 7f1818ff9640 1 -- 172.21.15.161:0/2260149689 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f180400ccf0 con 0x7f181c0d01b0 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.724+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f181c1406e0 con 0x7f181c0d01b0 2024-03-23T14:36:33.724 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.725+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f181c04f4e0 con 0x7f181c0d01b0 2024-03-23T14:36:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.725+0000 7f1818ff9640 1 -- 172.21.15.161:0/2260149689 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1804013470 con 0x7f181c0d01b0 2024-03-23T14:36:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.726+0000 7f1818ff9640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17fc06b810 0x7f17fc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.726+0000 7f1818ff9640 1 -- 172.21.15.161:0/2260149689 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f180408a380 con 0x7f181c0d01b0 2024-03-23T14:36:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.726+0000 7f181affd640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17fc06b810 0x7f17fc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.726+0000 7f181affd640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17fc06b810 0x7f17fc06dcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f181c154040 tx=0x7f180c007b70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:33.725 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.730+0000 7f1818ff9640 1 -- 172.21.15.161:0/2260149689 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1804054cc0 con 0x7f181c0d01b0 2024-03-23T14:36:33.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.902+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f181c0cc070 con 0x7f181c0d01b0 2024-03-23T14:36:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.903+0000 7f1818ff9640 1 -- 172.21.15.161:0/2260149689 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) ==== 100+0+11 (secure 0 0 0) 0x7f1804058380 con 0x7f181c0d01b0 2024-03-23T14:36:33.899 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:36:33.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17fc06b810 msgr2=0x7f17fc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17fc06b810 0x7f17fc06dcd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f181c154040 tx=0x7f180c007b70 comp rx=0 tx=0).stop 2024-03-23T14:36:33.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f181c0d01b0 msgr2=0x7f181c1558a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:33.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f181c0d01b0 0x7f181c1558a0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f180400e870 tx=0x7f180400ed30 comp rx=0 tx=0).stop 2024-03-23T14:36:33.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 shutdown_connections 2024-03-23T14:36:33.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f17fc06b810 0x7f17fc06dcd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f181c1552f0 0x7f181c155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f181c0d01b0 0x7f181c1558a0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 --2- 172.21.15.161:0/2260149689 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f181c0b2aa0 0x7f181c14e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:33.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 >> 172.21.15.161:0/2260149689 conn(0x7f181c06b260 msgr2=0x7f181c06ded0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:33.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 shutdown_connections 2024-03-23T14:36:33.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:33.905+0000 7f182193e640 1 -- 172.21.15.161:0/2260149689 wait complete. 2024-03-23T14:36:33.912 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-23T14:36:34.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.019+0000 7fe862a9a640 1 Processor -- start 2024-03-23T14:36:34.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.019+0000 7fe862a9a640 1 -- start start 2024-03-23T14:36:34.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe862a9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b43f0 0x7fe85c15b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe862a9a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe85c0d05f0 0x7fe85c15b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe862a9a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe85c15c1f0 con 0x7fe85c0b4010 2024-03-23T14:36:34.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe862a9a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fe85c06ef20 con 0x7fe85c0d05f0 2024-03-23T14:36:34.017 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe862a9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe85c15f730 con 0x7fe85c0b43f0 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe860836640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe85c0b4010 0x7fe85c0cfee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45126/0 (socket says 172.21.15.161:45126) 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe860836640 1 -- 172.21.15.161:0/3889227643 learned_addr learned my addr 172.21.15.161:0/3889227643 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe861037640 1 --2- 172.21.15.161:0/3889227643 >> v2:172.21.15.183:3300/0 conn(0x7fe85c0d05f0 0x7fe85c15b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe85bfff640 1 --2- 172.21.15.161:0/3889227643 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b43f0 0x7fe85c15b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe85b7fe640 1 -- 172.21.15.161:0/3889227643 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2060969231 0 0) 0x7fe85c15c1f0 con 0x7fe85c0b4010 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe85b7fe640 1 -- 172.21.15.161:0/3889227643 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe844003520 con 0x7fe85c0b4010 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe861037640 1 -- 172.21.15.161:0/3889227643 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b43f0 msgr2=0x7fe85c15b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe861037640 1 --2- 172.21.15.161:0/3889227643 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b43f0 0x7fe85c15b480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe861037640 1 -- 172.21.15.161:0/3889227643 >> v1:172.21.15.161:6789/0 conn(0x7fe85c0b4010 legacy=0x7fe85c0cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.020+0000 7fe861037640 1 -- 172.21.15.161:0/3889227643 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe85c15e970 con 0x7fe85c0d05f0 2024-03-23T14:36:34.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.021+0000 7fe861037640 1 --2- 172.21.15.161:0/3889227643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 0x7fe85c15b9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe854009dc0 tx=0x7fe85402be00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b4077471dadd7d7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.021+0000 7fe85b7fe640 1 -- 172.21.15.161:0/3889227643 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85402cbb0 con 0x7fe85c0d05f0 2024-03-23T14:36:34.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.021+0000 7fe85b7fe640 1 -- 172.21.15.161:0/3889227643 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85402cd10 con 0x7fe85c0d05f0 2024-03-23T14:36:34.019 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.021+0000 7fe85b7fe640 1 -- 172.21.15.161:0/3889227643 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe854034640 con 0x7fe85c0d05f0 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.022+0000 7fe862a9a640 1 -- 172.21.15.161:0/3889227643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 msgr2=0x7fe85c15b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.022+0000 7fe862a9a640 1 --2- 172.21.15.161:0/3889227643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 0x7fe85c15b9c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe854009dc0 tx=0x7fe85402be00 comp rx=0 tx=0).stop 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.022+0000 7fe862a9a640 1 -- 172.21.15.161:0/3889227643 shutdown_connections 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.022+0000 7fe862a9a640 1 --2- 172.21.15.161:0/3889227643 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 0x7fe85c15b9c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.022+0000 7fe862a9a640 1 --2- 172.21.15.161:0/3889227643 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b43f0 0x7fe85c15b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.022+0000 7fe862a9a640 1 -- 172.21.15.161:0/3889227643 >> 172.21.15.161:0/3889227643 conn(0x7fe85c06b4a0 msgr2=0x7fe85c06b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.022+0000 7fe862a9a640 1 -- 172.21.15.161:0/3889227643 shutdown_connections 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.022+0000 7fe862a9a640 1 -- 172.21.15.161:0/3889227643 wait complete. 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.023+0000 7fe862a9a640 1 Processor -- start 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.023+0000 7fe862a9a640 1 -- start start 2024-03-23T14:36:34.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe862a9a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b4010 0x7fe85c1f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe862a9a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe85c0b43f0 0x7fe85c15c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe862a9a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 0x7fe85c147430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe862a9a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c15d830 con 0x7fe85c0b43f0 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe862a9a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe85c15c8e0 con 0x7fe85c0d05f0 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe862a9a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe85c15cbb0 con 0x7fe85c0b4010 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe85c0b43f0 0x7fe85c15c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe85bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe85c0b43f0 0x7fe85c15c3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50660/0 (socket says 172.21.15.161:50660) 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe85bfff640 1 -- 172.21.15.161:0/2698148668 learned_addr learned my addr 172.21.15.161:0/2698148668 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe861037640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 0x7fe85c147430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe860836640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b4010 0x7fe85c1f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe85bfff640 1 -- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b4010 msgr2=0x7fe85c1f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe85bfff640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b4010 0x7fe85c1f28b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.023 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe85bfff640 1 -- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 msgr2=0x7fe85c147430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.024+0000 7fe85bfff640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 0x7fe85c147430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.025+0000 7fe85bfff640 1 -- 172.21.15.161:0/2698148668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe854009a70 con 0x7fe85c0b43f0 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.025+0000 7fe860836640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b4010 0x7fe85c1f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.025+0000 7fe85bfff640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe85c0b43f0 0x7fe85c15c3a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe85000e980 tx=0x7fe85000ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.025+0000 7fe8597fa640 1 -- 172.21.15.161:0/2698148668 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe850009670 con 0x7fe85c0b43f0 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.025+0000 7fe8597fa640 1 -- 172.21.15.161:0/2698148668 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe85000ca00 con 0x7fe85c0b43f0 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.025+0000 7fe8597fa640 1 -- 172.21.15.161:0/2698148668 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe85000cd90 con 0x7fe85c0b43f0 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.025+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe85c15ce90 con 0x7fe85c0b43f0 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.025+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe85c15d450 con 0x7fe85c0b43f0 2024-03-23T14:36:34.024 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.026+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe85c073570 con 0x7fe85c0b43f0 2024-03-23T14:36:34.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.026+0000 7fe8597fa640 1 -- 172.21.15.161:0/2698148668 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe850013470 con 0x7fe85c0b43f0 2024-03-23T14:36:34.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.027+0000 7fe8597fa640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe84806b810 0x7fe84806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.027+0000 7fe8597fa640 1 -- 172.21.15.161:0/2698148668 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fe85008a460 con 0x7fe85c0b43f0 2024-03-23T14:36:34.025 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.027+0000 7fe860836640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe84806b810 0x7fe84806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.026 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.027+0000 7fe860836640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe84806b810 0x7fe84806dcd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe84c005880 tx=0x7fe84c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.027 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.031+0000 7fe8597fa640 1 -- 172.21.15.161:0/2698148668 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe850054e30 con 0x7fe85c0b43f0 2024-03-23T14:36:34.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.202+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fe85c04f550 con 0x7fe85c0b43f0 2024-03-23T14:36:34.198 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.202+0000 7fe8597fa640 1 -- 172.21.15.161:0/2698148668 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v36) ==== 97+0+11 (secure 0 0 0) 0x7fe8500584f0 con 0x7fe85c0b43f0 2024-03-23T14:36:34.198 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:34.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe84806b810 msgr2=0x7fe84806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe84806b810 0x7fe84806dcd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe84c005880 tx=0x7fe84c009040 comp rx=0 tx=0).stop 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe85c0b43f0 msgr2=0x7fe85c15c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe85c0b43f0 0x7fe85c15c3a0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7fe85000e980 tx=0x7fe85000ee40 comp rx=0 tx=0).stop 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 shutdown_connections 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe84806b810 0x7fe84806dcd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe85c0d05f0 0x7fe85c147430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe85c0b43f0 0x7fe85c15c3a0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 --2- 172.21.15.161:0/2698148668 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe85c0b4010 0x7fe85c1f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 >> 172.21.15.161:0/2698148668 conn(0x7fe85c06b4a0 msgr2=0x7fe85c06b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 shutdown_connections 2024-03-23T14:36:34.201 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.204+0000 7fe862a9a640 1 -- 172.21.15.161:0/2698148668 wait complete. 2024-03-23T14:36:34.211 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:36:34.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.317+0000 7fceb4e3b640 1 Processor -- start 2024-03-23T14:36:34.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.317+0000 7fceb4e3b640 1 -- start start 2024-03-23T14:36:34.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fceb4e3b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb015b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fceb4e3b640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb015be70 con 0x7fceb006eca0 2024-03-23T14:36:34.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fceb4e3b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb015c150 con 0x7fceb00724e0 2024-03-23T14:36:34.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fceb4e3b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fceb015c430 con 0x7fceb0072100 2024-03-23T14:36:34.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fceaed9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fceb006eca0 0x7fceb00b3470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45130/0 (socket says 172.21.15.161:45130) 2024-03-23T14:36:34.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fceaed9d640 1 -- 172.21.15.161:0/359811810 learned_addr learned my addr 172.21.15.161:0/359811810 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:34.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fceae59c640 1 --2- 172.21.15.161:0/359811810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb015b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fcead59a640 1 -- 172.21.15.161:0/359811810 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129955320 0 0) 0x7fceb015be70 con 0x7fceb006eca0 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fcead59a640 1 -- 172.21.15.161:0/359811810 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce98003520 con 0x7fceb006eca0 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fcead59a640 1 -- 172.21.15.161:0/359811810 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 865384766 0 0) 0x7fceb015c150 con 0x7fceb00724e0 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fcead59a640 1 -- 172.21.15.161:0/359811810 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceb015be70 con 0x7fceb00724e0 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.318+0000 7fceae59c640 1 -- 172.21.15.161:0/359811810 >> v1:172.21.15.183:6789/0 conn(0x7fceb00724e0 legacy=0x7fceb006e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.319+0000 7fceae59c640 1 -- 172.21.15.161:0/359811810 >> v1:172.21.15.161:6789/0 conn(0x7fceb006eca0 legacy=0x7fceb00b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.319+0000 7fceae59c640 1 -- 172.21.15.161:0/359811810 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb00b2c70 con 0x7fceb0072100 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.319+0000 7fceae59c640 1 --2- 172.21.15.161:0/359811810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb015b3f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcea00097d0 tx=0x7fcea002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfafbe45d72da40c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.319+0000 7fcead59a640 1 -- 172.21.15.161:0/359811810 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0031890 con 0x7fceb0072100 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.319+0000 7fcead59a640 1 -- 172.21.15.161:0/359811810 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea00319f0 con 0x7fceb0072100 2024-03-23T14:36:34.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.319+0000 7fcead59a640 1 -- 172.21.15.161:0/359811810 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea0031d60 con 0x7fceb0072100 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.320+0000 7fceb4e3b640 1 -- 172.21.15.161:0/359811810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 msgr2=0x7fceb015b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.320+0000 7fceb4e3b640 1 --2- 172.21.15.161:0/359811810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb015b3f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcea00097d0 tx=0x7fcea002b930 comp rx=0 tx=0).stop 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.320+0000 7fceb4e3b640 1 -- 172.21.15.161:0/359811810 shutdown_connections 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.320+0000 7fceb4e3b640 1 --2- 172.21.15.161:0/359811810 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb015b3f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.320+0000 7fceb4e3b640 1 -- 172.21.15.161:0/359811810 >> 172.21.15.161:0/359811810 conn(0x7fceb00b17f0 msgr2=0x7fceb00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.320+0000 7fceb4e3b640 1 -- 172.21.15.161:0/359811810 shutdown_connections 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.320+0000 7fceb4e3b640 1 -- 172.21.15.161:0/359811810 wait complete. 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.321+0000 7fceb4e3b640 1 Processor -- start 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.321+0000 7fceb4e3b640 1 -- start start 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceb4e3b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb006eca0 0x7fceb01f1cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceb4e3b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb0148380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceb4e3b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb00724e0 0x7fceb00a94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceb4e3b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fceb0158300 con 0x7fceb00724e0 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceb4e3b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fceb01585e0 con 0x7fceb006eca0 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceb4e3b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fceb01588c0 con 0x7fceb0072100 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceaed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb00724e0 0x7fceb00a94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceaed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb00724e0 0x7fceb00a94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50668/0 (socket says 172.21.15.161:50668) 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceaed9d640 1 -- 172.21.15.161:0/2010984976 learned_addr learned my addr 172.21.15.161:0/2010984976 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceae59c640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb006eca0 0x7fceb01f1cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceadd9b640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb0148380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceaed9d640 1 -- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 msgr2=0x7fceb0148380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceaed9d640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb0148380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceaed9d640 1 -- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb006eca0 msgr2=0x7fceb01f1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceaed9d640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb006eca0 0x7fceb01f1cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.322+0000 7fceaed9d640 1 -- 172.21.15.161:0/2010984976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea0009400 con 0x7fceb00724e0 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.323+0000 7fceaed9d640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb00724e0 0x7fceb00a94a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcea800caf0 tx=0x7fcea800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.323+0000 7fce9effd640 1 -- 172.21.15.161:0/2010984976 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea800ac40 con 0x7fceb00724e0 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.323+0000 7fce9effd640 1 -- 172.21.15.161:0/2010984976 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea800ada0 con 0x7fceb00724e0 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.323+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb0158ba0 con 0x7fceb00724e0 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.323+0000 7fce9effd640 1 -- 172.21.15.161:0/2010984976 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8012620 con 0x7fceb00724e0 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.323+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb01548e0 con 0x7fceb00724e0 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.324+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb01590c0 con 0x7fceb00724e0 2024-03-23T14:36:34.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.324+0000 7fce9effd640 1 -- 172.21.15.161:0/2010984976 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcea8007070 con 0x7fceb00724e0 2024-03-23T14:36:34.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.325+0000 7fce9effd640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce9006b810 0x7fce9006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.325+0000 7fce9effd640 1 -- 172.21.15.161:0/2010984976 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fcea808a230 con 0x7fceb00724e0 2024-03-23T14:36:34.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.325+0000 7fceae59c640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce9006b810 0x7fce9006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.323 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.325+0000 7fceae59c640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce9006b810 0x7fce9006dcd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcea0036000 tx=0x7fcea003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.325 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.329+0000 7fce9effd640 1 -- 172.21.15.161:0/2010984976 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fceb01590c0 con 0x7fceb00724e0 2024-03-23T14:36:34.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.511+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fceb01590c0 con 0x7fceb00724e0 2024-03-23T14:36:34.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.511+0000 7fce9effd640 1 -- 172.21.15.161:0/2010984976 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) ==== 80+0+0 (secure 0 0 0) 0x7fceb01590c0 con 0x7fceb00724e0 2024-03-23T14:36:34.507 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:36:34.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce9006b810 msgr2=0x7fce9006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce9006b810 0x7fce9006dcd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fcea0036000 tx=0x7fcea003c040 comp rx=0 tx=0).stop 2024-03-23T14:36:34.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb00724e0 msgr2=0x7fceb00a94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb00724e0 0x7fceb00a94a0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7fcea800caf0 tx=0x7fcea800a060 comp rx=0 tx=0).stop 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 shutdown_connections 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fce9006b810 0x7fce9006dcd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb00724e0 0x7fceb00a94a0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb0072100 0x7fceb0148380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 --2- 172.21.15.161:0/2010984976 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb006eca0 0x7fceb01f1cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 >> 172.21.15.161:0/2010984976 conn(0x7fceb00b17f0 msgr2=0x7fceb00cf0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 shutdown_connections 2024-03-23T14:36:34.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.513+0000 7fceb4e3b640 1 -- 172.21.15.161:0/2010984976 wait complete. 2024-03-23T14:36:34.520 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:34.520 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:34.535 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:34.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.679+0000 7f8c25eeb640 1 Processor -- start 2024-03-23T14:36:34.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.679+0000 7f8c25eeb640 1 -- start start 2024-03-23T14:36:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.679+0000 7f8c25eeb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c200724e0 0x7f8c2006e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.679+0000 7f8c25eeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 0x7f8c200b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.679+0000 7f8c25eeb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c2015be70 con 0x7f8c20072100 2024-03-23T14:36:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.679+0000 7f8c25eeb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8c2006ef80 con 0x7f8c200724e0 2024-03-23T14:36:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.679+0000 7f8c25eeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8c2015c430 con 0x7f8c2006ead0 2024-03-23T14:36:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c200724e0 0x7f8c2006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1f7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8c20072100 0x7f8c2015b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45146/0 (socket says 172.21.15.161:45146) 2024-03-23T14:36:34.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1f7fe640 1 -- 172.21.15.161:0/4017054189 learned_addr learned my addr 172.21.15.161:0/4017054189 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:34.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1ffff640 1 --2- 172.21.15.161:0/4017054189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 0x7f8c200b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1effd640 1 -- 172.21.15.161:0/4017054189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 msgr2=0x7f8c200b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:34.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1effd640 1 --2- 172.21.15.161:0/4017054189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 0x7f8c200b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1effd640 1 -- 172.21.15.161:0/4017054189 >> v1:172.21.15.161:6789/0 conn(0x7f8c20072100 legacy=0x7f8c2015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1effd640 1 -- 172.21.15.161:0/4017054189 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2015c150 con 0x7f8c200724e0 2024-03-23T14:36:34.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.680+0000 7f8c1e7fc640 1 -- 172.21.15.161:0/4017054189 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1279054517 0 0) 0x7f8c2015be70 con 0x7f8c20072100 2024-03-23T14:36:34.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c1effd640 1 --2- 172.21.15.161:0/4017054189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c2006e590 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8c140097a0 tx=0x7f8c1402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14aced88f78a1b76 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c1e7fc640 1 -- 172.21.15.161:0/4017054189 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c14031830 con 0x7f8c200724e0 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c1e7fc640 1 -- 172.21.15.161:0/4017054189 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c14031990 con 0x7f8c200724e0 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c1e7fc640 1 -- 172.21.15.161:0/4017054189 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c14031d20 con 0x7f8c200724e0 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c25eeb640 1 -- 172.21.15.161:0/4017054189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 msgr2=0x7f8c2006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/4017054189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c2006e590 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f8c140097a0 tx=0x7f8c1402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c25eeb640 1 -- 172.21.15.161:0/4017054189 shutdown_connections 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/4017054189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 0x7f8c200b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/4017054189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c2006e590 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c25eeb640 1 -- 172.21.15.161:0/4017054189 >> 172.21.15.161:0/4017054189 conn(0x7f8c200b17f0 msgr2=0x7f8c200b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c25eeb640 1 -- 172.21.15.161:0/4017054189 shutdown_connections 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.681+0000 7f8c25eeb640 1 -- 172.21.15.161:0/4017054189 wait complete. 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.682+0000 7f8c25eeb640 1 Processor -- start 2024-03-23T14:36:34.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.683+0000 7f8c25eeb640 1 -- start start 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.683+0000 7f8c25eeb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 0x7f8c20150a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.683+0000 7f8c25eeb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c20072100 0x7f8c20152670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.683+0000 7f8c25eeb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c201425c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.683+0000 7f8c25eeb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c201f1b60 con 0x7f8c200724e0 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.683+0000 7f8c25eeb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8c20151130 con 0x7f8c20072100 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.683+0000 7f8c25eeb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8c20151410 con 0x7f8c2006ead0 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.683+0000 7f8c1ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c201425c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c201425c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50690/0 (socket says 172.21.15.161:50690) 2024-03-23T14:36:34.682 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1ffff640 1 -- 172.21.15.161:0/891860780 learned_addr learned my addr 172.21.15.161:0/891860780 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1f7fe640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 0x7f8c20150a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1effd640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c20072100 0x7f8c20152670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1ffff640 1 -- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 msgr2=0x7f8c20150a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1ffff640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 0x7f8c20150a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1ffff640 1 -- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c20072100 msgr2=0x7f8c20152670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1ffff640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c20072100 0x7f8c20152670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1ffff640 1 -- 172.21.15.161:0/891860780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c14009400 con 0x7f8c200724e0 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c1ffff640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c201425c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800e7f0 tx=0x7f8c1800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c07fff640 1 -- 172.21.15.161:0/891860780 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c18009460 con 0x7f8c200724e0 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c07fff640 1 -- 172.21.15.161:0/891860780 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c1800c730 con 0x7f8c200724e0 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c07fff640 1 -- 172.21.15.161:0/891860780 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1800cae0 con 0x7f8c200724e0 2024-03-23T14:36:34.683 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c201f1dd0 con 0x7f8c200724e0 2024-03-23T14:36:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.684+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c201516f0 con 0x7f8c200724e0 2024-03-23T14:36:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.685+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c200735e0 con 0x7f8c200724e0 2024-03-23T14:36:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.685+0000 7f8c07fff640 1 -- 172.21.15.161:0/891860780 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8c1800cc40 con 0x7f8c200724e0 2024-03-23T14:36:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.686+0000 7f8c07fff640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c0c06b810 0x7f8c0c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.686+0000 7f8c07fff640 1 -- 172.21.15.161:0/891860780 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f8c1808a690 con 0x7f8c200724e0 2024-03-23T14:36:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.686+0000 7f8c1f7fe640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c0c06b810 0x7f8c0c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.684 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.686+0000 7f8c1f7fe640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c0c06b810 0x7f8c0c06dcd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8c100012e0 tx=0x7f8c10001700 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.690+0000 7f8c07fff640 1 -- 172.21.15.161:0/891860780 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8c18054fd0 con 0x7f8c200724e0 2024-03-23T14:36:34.858 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.862+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8c2015e3b0 con 0x7f8c200724e0 2024-03-23T14:36:34.859 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.863+0000 7f8c07fff640 1 -- 172.21.15.161:0/891860780 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7f8c18058690 con 0x7f8c200724e0 2024-03-23T14:36:34.859 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:34.859 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c0c06b810 msgr2=0x7f8c0c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c0c06b810 0x7f8c0c06dcd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8c100012e0 tx=0x7f8c10001700 comp rx=0 tx=0).stop 2024-03-23T14:36:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 msgr2=0x7f8c201425c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.861 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c201425c0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f8c1800e7f0 tx=0x7f8c1800ecb0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 shutdown_connections 2024-03-23T14:36:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c0c06b810 0x7f8c0c06dcd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c200724e0 0x7f8c201425c0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c20072100 0x7f8c20152670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 --2- 172.21.15.161:0/891860780 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c2006ead0 0x7f8c20150a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 >> 172.21.15.161:0/891860780 conn(0x7f8c200b17f0 msgr2=0x7f8c2006dd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 shutdown_connections 2024-03-23T14:36:34.862 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.865+0000 7f8c25eeb640 1 -- 172.21.15.161:0/891860780 wait complete. 2024-03-23T14:36:34.872 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:04.688180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:05.241774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-23T14:36:18.738957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:34.873 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:34.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.979+0000 7f96e5f5e640 1 Processor -- start 2024-03-23T14:36:34.975 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.979+0000 7f96e5f5e640 1 -- start start 2024-03-23T14:36:34.976 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96e5f5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d1270 0x7f96e00d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96e5f5e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f96e00d29d0 0x7f96e00b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96e5f5e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96e0161c00 con 0x7f96e0161820 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96e5f5e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f96e00d2e20 con 0x7f96e00d29d0 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96e5f5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f96e00cea30 con 0x7f96e00d1270 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96dffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f96e00d29d0 0x7f96e00b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96deffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d1270 0x7f96e00d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96df7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f96e0161820 0x7f96e00d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:59550/0 (socket says 172.21.15.161:59550) 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96deffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d1270 0x7f96e00d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:57074/0 (socket says 172.21.15.161:57074) 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96dffff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f96e00d29d0 0x7f96e00b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:57312/0 (socket says 172.21.15.161:57312) 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.980+0000 7f96df7fe640 1 -- 172.21.15.161:0/694240987 learned_addr learned my addr 172.21.15.161:0/694240987 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.981+0000 7f96deffd640 1 -- 172.21.15.161:0/694240987 >> v2:172.21.15.183:3300/0 conn(0x7f96e00d29d0 msgr2=0x7f96e00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:34.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.981+0000 7f96deffd640 1 --2- 172.21.15.161:0/694240987 >> v2:172.21.15.183:3300/0 conn(0x7f96e00d29d0 0x7f96e00b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.981+0000 7f96deffd640 1 -- 172.21.15.161:0/694240987 >> v1:172.21.15.183:6789/0 conn(0x7f96e0161820 legacy=0x7f96e00d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.981+0000 7f96deffd640 1 -- 172.21.15.161:0/694240987 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96e00cfd70 con 0x7f96e00d1270 2024-03-23T14:36:34.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.981+0000 7f96dffff640 1 --2- 172.21.15.161:0/694240987 >> v2:172.21.15.183:3300/0 conn(0x7f96e00d29d0 0x7f96e00b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:34.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96deffd640 1 --2- 172.21.15.161:0/694240987 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d1270 0x7f96e00d2490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f96d40097a0 tx=0x7f96d402b8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db79924026feb9c3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96de7fc640 1 -- 172.21.15.161:0/694240987 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4031830 con 0x7f96e00d1270 2024-03-23T14:36:34.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96de7fc640 1 -- 172.21.15.161:0/694240987 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d4031990 con 0x7f96e00d1270 2024-03-23T14:36:34.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96de7fc640 1 -- 172.21.15.161:0/694240987 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d4031d20 con 0x7f96e00d1270 2024-03-23T14:36:34.979 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96e5f5e640 1 -- 172.21.15.161:0/694240987 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d1270 msgr2=0x7f96e00d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/694240987 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d1270 0x7f96e00d2490 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f96d40097a0 tx=0x7f96d402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96e5f5e640 1 -- 172.21.15.161:0/694240987 shutdown_connections 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/694240987 >> v2:172.21.15.183:3300/0 conn(0x7f96e00d29d0 0x7f96e00b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/694240987 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d1270 0x7f96e00d2490 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96e5f5e640 1 -- 172.21.15.161:0/694240987 >> 172.21.15.161:0/694240987 conn(0x7f96e009d190 msgr2=0x7f96e00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96e5f5e640 1 -- 172.21.15.161:0/694240987 shutdown_connections 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.982+0000 7f96e5f5e640 1 -- 172.21.15.161:0/694240987 wait complete. 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.983+0000 7f96e5f5e640 1 Processor -- start 2024-03-23T14:36:34.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96e5f5e640 1 -- start start 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96e5f5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e00d1270 0x7f96e00b8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96e5f5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d29d0 0x7f96e00b1e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96e5f5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e0161820 0x7f96e00cd580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96e5f5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00b7c20 con 0x7f96e00d1270 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96e5f5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f96e00ce070 con 0x7f96e0161820 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96e5f5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f96e00b7e50 con 0x7f96e00d29d0 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96df7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e00d1270 0x7f96e00b8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96df7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e00d1270 0x7f96e00b8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50700/0 (socket says 172.21.15.161:50700) 2024-03-23T14:36:34.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.984+0000 7f96df7fe640 1 -- 172.21.15.161:0/2377435459 learned_addr learned my addr 172.21.15.161:0/2377435459 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96deffd640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d29d0 0x7f96e00b1e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96dffff640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e0161820 0x7f96e00cd580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96df7fe640 1 -- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d29d0 msgr2=0x7f96e00b1e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96df7fe640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d29d0 0x7f96e00b1e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96df7fe640 1 -- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e0161820 msgr2=0x7f96e00cd580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96df7fe640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e0161820 0x7f96e00cd580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96df7fe640 1 -- 172.21.15.161:0/2377435459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96d8009a70 con 0x7f96e00d1270 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96deffd640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d29d0 0x7f96e00b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96dffff640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e0161820 0x7f96e00cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96df7fe640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e00d1270 0x7f96e00b8c10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f96d000c400 tx=0x7f96d000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96c7fff640 1 -- 172.21.15.161:0/2377435459 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d0005ce0 con 0x7f96e00d1270 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96c7fff640 1 -- 172.21.15.161:0/2377435459 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f96d0005e40 con 0x7f96e00d1270 2024-03-23T14:36:34.984 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96c7fff640 1 -- 172.21.15.161:0/2377435459 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96d00072d0 con 0x7f96e00d1270 2024-03-23T14:36:34.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96d4009400 con 0x7f96e00d1270 2024-03-23T14:36:34.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.985+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96e00b86f0 con 0x7f96e00d1270 2024-03-23T14:36:34.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.986+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96e004f4e0 con 0x7f96e00d1270 2024-03-23T14:36:34.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.987+0000 7f96c7fff640 1 -- 172.21.15.161:0/2377435459 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f96d00074f0 con 0x7f96e00d1270 2024-03-23T14:36:34.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.987+0000 7f96c7fff640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96cc06b810 0x7f96cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:34.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.987+0000 7f96c7fff640 1 -- 172.21.15.161:0/2377435459 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f96d0089850 con 0x7f96e00d1270 2024-03-23T14:36:34.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.987+0000 7f96deffd640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96cc06b810 0x7f96cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:34.985 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.987+0000 7f96deffd640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96cc06b810 0x7f96cc06dcd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f96d4036000 tx=0x7f96d403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:34.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:34.991+0000 7f96c7fff640 1 -- 172.21.15.161:0/2377435459 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f96d0054190 con 0x7f96e00d1270 2024-03-23T14:36:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.162+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f96e00cf840 con 0x7f96e00d1270 2024-03-23T14:36:35.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.162+0000 7f96c7fff640 1 -- 172.21.15.161:0/2377435459 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f96d00170a0 con 0x7f96e00d1270 2024-03-23T14:36:35.158 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96cc06b810 msgr2=0x7f96cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96cc06b810 0x7f96cc06dcd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f96d4036000 tx=0x7f96d403c040 comp rx=0 tx=0).stop 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e00d1270 msgr2=0x7f96e00b8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e00d1270 0x7f96e00b8c10 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7f96d000c400 tx=0x7f96d000c800 comp rx=0 tx=0).stop 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 shutdown_connections 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f96cc06b810 0x7f96cc06dcd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f96e0161820 0x7f96e00cd580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f96e00d29d0 0x7f96e00b1e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 --2- 172.21.15.161:0/2377435459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f96e00d1270 0x7f96e00b8c10 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 >> 172.21.15.161:0/2377435459 conn(0x7f96e009d190 msgr2=0x7f96e00d3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:35.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 shutdown_connections 2024-03-23T14:36:35.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.164+0000 7f96e5f5e640 1 -- 172.21.15.161:0/2377435459 wait complete. 2024-03-23T14:36:35.171 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:36:35.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.279+0000 7fd50ebf4640 1 Processor -- start 2024-03-23T14:36:35.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.279+0000 7fd50ebf4640 1 -- start start 2024-03-23T14:36:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.279+0000 7fd50ebf4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd508161750 0x7fd5080d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.279+0000 7fd50ebf4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd5080d1cb0 0x7fd5080d2b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.279+0000 7fd50ebf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd508161b30 0x7fd508162060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.279+0000 7fd50ebf4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd5080d1330 con 0x7fd508161750 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.279+0000 7fd50ebf4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fd5080d0720 con 0x7fd5080d1cb0 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.279+0000 7fd50ebf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd5080d0a00 con 0x7fd508161b30 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50c990640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd508161750 0x7fd5080d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50c990640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd508161750 0x7fd5080d25c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50710/0 (socket says 172.21.15.161:50710) 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50c990640 1 -- 172.21.15.161:0/1736070893 learned_addr learned my addr 172.21.15.161:0/1736070893 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50d191640 1 --2- 172.21.15.161:0/1736070893 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd508161b30 0x7fd508162060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd507fff640 1 --2- 172.21.15.161:0/1736070893 >> v2:172.21.15.183:3300/0 conn(0x7fd5080d1cb0 0x7fd5080d2b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50c990640 1 -- 172.21.15.161:0/1736070893 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd508161b30 msgr2=0x7fd508162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50c990640 1 --2- 172.21.15.161:0/1736070893 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd508161b30 0x7fd508162060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50c990640 1 -- 172.21.15.161:0/1736070893 >> v2:172.21.15.183:3300/0 conn(0x7fd5080d1cb0 msgr2=0x7fd5080d2b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50c990640 1 --2- 172.21.15.161:0/1736070893 >> v2:172.21.15.183:3300/0 conn(0x7fd5080d1cb0 0x7fd5080d2b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.280+0000 7fd50c990640 1 -- 172.21.15.161:0/1736070893 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5080d0ce0 con 0x7fd508161750 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50c990640 1 --2- 172.21.15.161:0/1736070893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080d25c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5080c7be0 tx=0x7fd4fc02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c56d7eac19920a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd5077fe640 1 -- 172.21.15.161:0/1736070893 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc031870 con 0x7fd508161750 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd5077fe640 1 -- 172.21.15.161:0/1736070893 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4fc0319d0 con 0x7fd508161750 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd5077fe640 1 -- 172.21.15.161:0/1736070893 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4fc031d60 con 0x7fd508161750 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1736070893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 msgr2=0x7fd5080d25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1736070893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080d25c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fd5080c7be0 tx=0x7fd4fc02b910 comp rx=0 tx=0).stop 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1736070893 shutdown_connections 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1736070893 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd508161b30 0x7fd508162060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1736070893 >> v2:172.21.15.183:3300/0 conn(0x7fd5080d1cb0 0x7fd5080d2b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1736070893 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080d25c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1736070893 >> 172.21.15.161:0/1736070893 conn(0x7fd50809d450 msgr2=0x7fd5080d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:35.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1736070893 shutdown_connections 2024-03-23T14:36:35.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.281+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1736070893 wait complete. 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.282+0000 7fd50ebf4640 1 Processor -- start 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.283+0000 7fd50ebf4640 1 -- start start 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.283+0000 7fd50ebf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd5080d1cb0 0x7fd5081f46b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.283+0000 7fd50ebf4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080b8800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.283+0000 7fd50ebf4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd508161b30 0x7fd5080ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.283+0000 7fd50ebf4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd5080cc280 con 0x7fd508161750 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.283+0000 7fd50ebf4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd5081f4bf0 con 0x7fd508161b30 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.283+0000 7fd50ebf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd5080cb600 con 0x7fd5080d1cb0 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080b8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080b8800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50718/0 (socket says 172.21.15.161:50718) 2024-03-23T14:36:35.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 -- 172.21.15.161:0/1890235617 learned_addr learned my addr 172.21.15.161:0/1890235617 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd50c990640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd5080d1cb0 0x7fd5081f46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd50d191640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd508161b30 0x7fd5080ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 -- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd5080d1cb0 msgr2=0x7fd5081f46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd5080d1cb0 0x7fd5081f46b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 -- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd508161b30 msgr2=0x7fd5080ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd508161b30 0x7fd5080ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 -- 172.21.15.161:0/1890235617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4fc0094d0 con 0x7fd508161750 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd507fff640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080b8800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400e860 tx=0x7fd4f400ed20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd5057fa640 1 -- 172.21.15.161:0/1890235617 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f40095f0 con 0x7fd508161750 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd5057fa640 1 -- 172.21.15.161:0/1890235617 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd4f400c940 con 0x7fd508161750 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5080cb8e0 con 0x7fd508161750 2024-03-23T14:36:35.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd5057fa640 1 -- 172.21.15.161:0/1890235617 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4f400ccd0 con 0x7fd508161750 2024-03-23T14:36:35.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.284+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5080cbea0 con 0x7fd508161750 2024-03-23T14:36:35.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.285+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5080a2030 con 0x7fd508161750 2024-03-23T14:36:35.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.286+0000 7fd5057fa640 1 -- 172.21.15.161:0/1890235617 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd4f4013470 con 0x7fd508161750 2024-03-23T14:36:35.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.286+0000 7fd5057fa640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd4ec06b810 0x7fd4ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.286+0000 7fd5057fa640 1 -- 172.21.15.161:0/1890235617 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fd4f408a4a0 con 0x7fd508161750 2024-03-23T14:36:35.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.286+0000 7fd50c990640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd4ec06b810 0x7fd4ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.287+0000 7fd50c990640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd4ec06b810 0x7fd4ec06dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc036000 tx=0x7fd4fc03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.290+0000 7fd5057fa640 1 -- 172.21.15.161:0/1890235617 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4f4054e70 con 0x7fd508161750 2024-03-23T14:36:35.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.459+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd50804f550 con 0x7fd508161750 2024-03-23T14:36:35.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.460+0000 7fd5057fa640 1 -- 172.21.15.161:0/1890235617 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) ==== 104+0+11 (secure 0 0 0) 0x7fd4f4058530 con 0x7fd508161750 2024-03-23T14:36:35.456 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd4ec06b810 msgr2=0x7fd4ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd4ec06b810 0x7fd4ec06dcd0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd4fc036000 tx=0x7fd4fc03c040 comp rx=0 tx=0).stop 2024-03-23T14:36:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 msgr2=0x7fd5080b8800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080b8800 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fd4f400e860 tx=0x7fd4f400ed20 comp rx=0 tx=0).stop 2024-03-23T14:36:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 shutdown_connections 2024-03-23T14:36:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd4ec06b810 0x7fd4ec06dcd0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd508161b30 0x7fd5080ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd508161750 0x7fd5080b8800 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 --2- 172.21.15.161:0/1890235617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd5080d1cb0 0x7fd5081f46b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 >> 172.21.15.161:0/1890235617 conn(0x7fd50809d450 msgr2=0x7fd508160980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 shutdown_connections 2024-03-23T14:36:35.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.462+0000 7fd50ebf4640 1 -- 172.21.15.161:0/1890235617 wait complete. 2024-03-23T14:36:35.469 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:36:35.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.576+0000 7fd6f31dd640 1 Processor -- start 2024-03-23T14:36:35.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.576+0000 7fd6f31dd640 1 -- start start 2024-03-23T14:36:35.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f31dd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 0x7fd6ec0cdf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f31dd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd6ec0d0380 0x7fd6ec0b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f31dd640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6ec0cf200 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f31dd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd6ec15d270 con 0x7fd6ec0d0380 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f31dd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ec0ce5f0 con 0x7fd6ec145570 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f177a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd6ec0d0380 0x7fd6ec0b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f177a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd6ec0d0380 0x7fd6ec0b2b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50724/0 (socket says 172.21.15.161:50724) 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f177a640 1 -- 172.21.15.161:0/1900076305 learned_addr learned my addr 172.21.15.161:0/1900076305 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f0f79640 1 --2- 172.21.15.161:0/1900076305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 0x7fd6ec0cdf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f177a640 1 -- 172.21.15.161:0/1900076305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 msgr2=0x7fd6ec0cdf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f177a640 1 --2- 172.21.15.161:0/1900076305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 0x7fd6ec0cdf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f177a640 1 -- 172.21.15.161:0/1900076305 >> v1:172.21.15.183:6789/0 conn(0x7fd6ec15cdf0 legacy=0x7fd6ec0cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.577+0000 7fd6f177a640 1 -- 172.21.15.161:0/1900076305 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6ec0ce8d0 con 0x7fd6ec0d0380 2024-03-23T14:36:35.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.578+0000 7fd6f177a640 1 --2- 172.21.15.161:0/1900076305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec0d0380 0x7fd6ec0b2b10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009870 tx=0x7fd6e402b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=295b3e9da56b5fbf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.578+0000 7fd6eb7fe640 1 -- 172.21.15.161:0/1900076305 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4030840 con 0x7fd6ec0d0380 2024-03-23T14:36:35.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.578+0000 7fd6eb7fe640 1 -- 172.21.15.161:0/1900076305 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e40309a0 con 0x7fd6ec0d0380 2024-03-23T14:36:35.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.578+0000 7fd6eb7fe640 1 -- 172.21.15.161:0/1900076305 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4030d30 con 0x7fd6ec0d0380 2024-03-23T14:36:35.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.579+0000 7fd6f31dd640 1 -- 172.21.15.161:0/1900076305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec0d0380 msgr2=0x7fd6ec0b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.579+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/1900076305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec0d0380 0x7fd6ec0b2b10 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009870 tx=0x7fd6e402b980 comp rx=0 tx=0).stop 2024-03-23T14:36:35.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.579+0000 7fd6f31dd640 1 -- 172.21.15.161:0/1900076305 shutdown_connections 2024-03-23T14:36:35.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.579+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/1900076305 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec0d0380 0x7fd6ec0b2b10 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.579+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/1900076305 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 0x7fd6ec0cdf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.579+0000 7fd6f31dd640 1 -- 172.21.15.161:0/1900076305 >> 172.21.15.161:0/1900076305 conn(0x7fd6ec06b260 msgr2=0x7fd6ec06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:35.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.579+0000 7fd6f31dd640 1 -- 172.21.15.161:0/1900076305 shutdown_connections 2024-03-23T14:36:35.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.579+0000 7fd6f31dd640 1 -- 172.21.15.161:0/1900076305 wait complete. 2024-03-23T14:36:35.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.580+0000 7fd6f31dd640 1 Processor -- start 2024-03-23T14:36:35.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.580+0000 7fd6f31dd640 1 -- start start 2024-03-23T14:36:35.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f31dd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6ec0d0380 0x7fd6ec138370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f31dd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 0x7fd6ec13b340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f31dd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec15cdf0 0x7fd6ec13c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f31dd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ec13a2a0 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f31dd640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd6ec13a580 con 0x7fd6ec0d0380 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f31dd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd6ec13a860 con 0x7fd6ec145570 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f177a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec15cdf0 0x7fd6ec13c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f0f79640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6ec0d0380 0x7fd6ec138370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f177a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec15cdf0 0x7fd6ec13c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50738/0 (socket says 172.21.15.161:50738) 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6ebfff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 0x7fd6ec13b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f177a640 1 -- 172.21.15.161:0/2793003042 learned_addr learned my addr 172.21.15.161:0/2793003042 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f177a640 1 -- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 msgr2=0x7fd6ec13b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f177a640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 0x7fd6ec13b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f177a640 1 -- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6ec0d0380 msgr2=0x7fd6ec138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f177a640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6ec0d0380 0x7fd6ec138370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.581+0000 7fd6f177a640 1 -- 172.21.15.161:0/2793003042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6e40094d0 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.582+0000 7fd6f177a640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec15cdf0 0x7fd6ec13c230 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009870 tx=0x7fd6e4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.582+0000 7fd6e97fa640 1 -- 172.21.15.161:0/2793003042 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4039c90 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.582+0000 7fd6e97fa640 1 -- 172.21.15.161:0/2793003042 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd6e4039e10 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.582+0000 7fd6e97fa640 1 -- 172.21.15.161:0/2793003042 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6e4043640 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.582+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6ec13ab40 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.582+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6ec1388b0 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.583+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd6ec13b060 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.583+0000 7fd6e97fa640 1 -- 172.21.15.161:0/2793003042 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd6e40437a0 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.583+0000 7fd6e97fa640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd6cc06b810 0x7fd6cc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.584+0000 7fd6e97fa640 1 -- 172.21.15.161:0/2793003042 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fd6e403e070 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.584+0000 7fd6f0f79640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd6cc06b810 0x7fd6cc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.584+0000 7fd6f0f79640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd6cc06b810 0x7fd6cc06dcd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0113e0 tx=0x7fd6dc00b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.588+0000 7fd6e97fa640 1 -- 172.21.15.161:0/2793003042 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd6ec13b060 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.758+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd6ec13b060 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.758+0000 7fd6e97fa640 1 -- 172.21.15.161:0/2793003042 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) ==== 100+0+11 (secure 0 0 0) 0x7fd6ec13b060 con 0x7fd6ec15cdf0 2024-03-23T14:36:35.754 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:36:35.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd6cc06b810 msgr2=0x7fd6cc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd6cc06b810 0x7fd6cc06dcd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fd6dc0113e0 tx=0x7fd6dc00b040 comp rx=0 tx=0).stop 2024-03-23T14:36:35.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec15cdf0 msgr2=0x7fd6ec13c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec15cdf0 0x7fd6ec13c230 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fd6e4009870 tx=0x7fd6e4004730 comp rx=0 tx=0).stop 2024-03-23T14:36:35.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 shutdown_connections 2024-03-23T14:36:35.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd6cc06b810 0x7fd6cc06dcd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd6ec15cdf0 0x7fd6ec13c230 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd6ec145570 0x7fd6ec13b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 --2- 172.21.15.161:0/2793003042 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd6ec0d0380 0x7fd6ec138370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 >> 172.21.15.161:0/2793003042 conn(0x7fd6ec06b260 msgr2=0x7fd6ec142220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:35.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 shutdown_connections 2024-03-23T14:36:35.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.760+0000 7fd6f31dd640 1 -- 172.21.15.161:0/2793003042 wait complete. 2024-03-23T14:36:35.767 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-23T14:36:35.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.875+0000 7fecc31b5640 1 Processor -- start 2024-03-23T14:36:35.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.875+0000 7fecc31b5640 1 -- start start 2024-03-23T14:36:35.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc31b5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fecbc145520 0x7fecbc160380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc31b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc0cfee0 0x7fecbc0d0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc31b5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fecbc0d0880 con 0x7fecbc070c20 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc31b5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fecbc1608c0 con 0x7fecbc145520 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc31b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fecbc0ce410 con 0x7fecbc0cfee0 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc0f51640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fecbc145520 0x7fecbc160380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc1752640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fecbc070c20 0x7fecbc0710b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45152/0 (socket says 172.21.15.161:45152) 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc1752640 1 -- 172.21.15.161:0/1715192587 learned_addr learned my addr 172.21.15.161:0/1715192587 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecbbfff640 1 --2- 172.21.15.161:0/1715192587 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc0cfee0 0x7fecbc0d0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc0f51640 1 -- 172.21.15.161:0/1715192587 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc0cfee0 msgr2=0x7fecbc0d0340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc0f51640 1 --2- 172.21.15.161:0/1715192587 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc0cfee0 0x7fecbc0d0340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc0f51640 1 -- 172.21.15.161:0/1715192587 >> v1:172.21.15.161:6789/0 conn(0x7fecbc070c20 legacy=0x7fecbc0710b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecc0f51640 1 -- 172.21.15.161:0/1715192587 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc0ce6f0 con 0x7fecbc145520 2024-03-23T14:36:35.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.876+0000 7fecbb7fe640 1 -- 172.21.15.161:0/1715192587 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 290201559 0 0) 0x7fecbc0d0880 con 0x7fecbc070c20 2024-03-23T14:36:35.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.877+0000 7fecc0f51640 1 --2- 172.21.15.161:0/1715192587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc145520 0x7fecbc160380 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fecb00097a0 tx=0x7fecb002b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bb16b480b30a1ef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.877+0000 7fecbb7fe640 1 -- 172.21.15.161:0/1715192587 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb002e070 con 0x7fecbc145520 2024-03-23T14:36:35.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.877+0000 7fecbb7fe640 1 -- 172.21.15.161:0/1715192587 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fecb00315a0 con 0x7fecbc145520 2024-03-23T14:36:35.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.877+0000 7fecbb7fe640 1 -- 172.21.15.161:0/1715192587 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb0031930 con 0x7fecbc145520 2024-03-23T14:36:35.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.877+0000 7fecc31b5640 1 -- 172.21.15.161:0/1715192587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc145520 msgr2=0x7fecbc160380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.877+0000 7fecc31b5640 1 --2- 172.21.15.161:0/1715192587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc145520 0x7fecbc160380 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7fecb00097a0 tx=0x7fecb002b840 comp rx=0 tx=0).stop 2024-03-23T14:36:35.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.878+0000 7fecc31b5640 1 -- 172.21.15.161:0/1715192587 shutdown_connections 2024-03-23T14:36:35.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.878+0000 7fecc31b5640 1 --2- 172.21.15.161:0/1715192587 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc0cfee0 0x7fecbc0d0340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.878+0000 7fecc31b5640 1 --2- 172.21.15.161:0/1715192587 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc145520 0x7fecbc160380 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.878+0000 7fecc31b5640 1 -- 172.21.15.161:0/1715192587 >> 172.21.15.161:0/1715192587 conn(0x7fecbc06b4a0 msgr2=0x7fecbc06b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:35.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.878+0000 7fecc31b5640 1 -- 172.21.15.161:0/1715192587 shutdown_connections 2024-03-23T14:36:35.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.878+0000 7fecc31b5640 1 -- 172.21.15.161:0/1715192587 wait complete. 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.879+0000 7fecc31b5640 1 Processor -- start 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.879+0000 7fecc31b5640 1 -- start start 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc31b5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc070c20 0x7fecbc0b3b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc31b5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc0cfee0 0x7fecbc1f50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc31b5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecbc145520 0x7fecbc1f55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc31b5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1f6cc0 con 0x7fecbc0cfee0 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc31b5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1f5b20 con 0x7fecbc145520 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc31b5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fecbc1569f0 con 0x7fecbc070c20 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc0cfee0 0x7fecbc1f50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc0cfee0 0x7fecbc1f50a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50764/0 (socket says 172.21.15.161:50764) 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 -- 172.21.15.161:0/4279049561 learned_addr learned my addr 172.21.15.161:0/4279049561 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc0f51640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc070c20 0x7fecbc0b3b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc1752640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecbc145520 0x7fecbc1f55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 -- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc070c20 msgr2=0x7fecbc0b3b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc070c20 0x7fecbc0b3b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 -- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecbc145520 msgr2=0x7fecbc1f55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecbc145520 0x7fecbc1f55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 -- 172.21.15.161:0/4279049561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecb0009400 con 0x7fecbc0cfee0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc0f51640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc070c20 0x7fecbc0b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc1752640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecbc145520 0x7fecbc1f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecbbfff640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc0cfee0 0x7fecbc1f50a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7feca800e9b0 tx=0x7feca800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecb97fa640 1 -- 172.21.15.161:0/4279049561 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca8009670 con 0x7fecbc0cfee0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecb97fa640 1 -- 172.21.15.161:0/4279049561 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feca800c9a0 con 0x7fecbc0cfee0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecbc1560c0 con 0x7fecbc0cfee0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecb97fa640 1 -- 172.21.15.161:0/4279049561 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feca800cd50 con 0x7fecbc0cfee0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.880+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecbc156680 con 0x7fecbc0cfee0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.881+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fecbc0b4460 con 0x7fecbc0cfee0 2024-03-23T14:36:35.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.882+0000 7fecb97fa640 1 -- 172.21.15.161:0/4279049561 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feca8013470 con 0x7fecbc0cfee0 2024-03-23T14:36:35.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.882+0000 7fecb97fa640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feca006b840 0x7feca006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:35.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.882+0000 7fecb97fa640 1 -- 172.21.15.161:0/4279049561 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7feca808a580 con 0x7fecbc0cfee0 2024-03-23T14:36:35.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.882+0000 7fecc0f51640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feca006b840 0x7feca006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:35.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.883+0000 7fecc0f51640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feca006b840 0x7feca006dd00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fecb0036000 tx=0x7fecb003d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:35.882 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:35.886+0000 7fecb97fa640 1 -- 172.21.15.161:0/4279049561 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feca8054e10 con 0x7fecbc0cfee0 2024-03-23T14:36:36.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.059+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7fecbc073570 con 0x7fecbc0cfee0 2024-03-23T14:36:36.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.059+0000 7fecb97fa640 1 -- 172.21.15.161:0/4279049561 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v36) ==== 97+0+11 (secure 0 0 0) 0x7feca80584d0 con 0x7fecbc0cfee0 2024-03-23T14:36:36.055 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:36.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.061+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feca006b840 msgr2=0x7feca006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.061+0000 7fecc31b5640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feca006b840 0x7feca006dd00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fecb0036000 tx=0x7fecb003d040 comp rx=0 tx=0).stop 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.061+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc0cfee0 msgr2=0x7fecbc1f50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.061+0000 7fecc31b5640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc0cfee0 0x7fecbc1f50a0 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7feca800e9b0 tx=0x7feca800ee70 comp rx=0 tx=0).stop 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.062+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 shutdown_connections 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.062+0000 7fecc31b5640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feca006b840 0x7feca006dd00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.062+0000 7fecc31b5640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fecbc145520 0x7fecbc1f55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.062+0000 7fecc31b5640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fecbc0cfee0 0x7fecbc1f50a0 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.062+0000 7fecc31b5640 1 --2- 172.21.15.161:0/4279049561 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fecbc070c20 0x7fecbc0b3b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.062+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 >> 172.21.15.161:0/4279049561 conn(0x7fecbc06b4a0 msgr2=0x7fecbc0b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.062+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 shutdown_connections 2024-03-23T14:36:36.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.062+0000 7fecc31b5640 1 -- 172.21.15.161:0/4279049561 wait complete. 2024-03-23T14:36:36.068 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:36:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.174+0000 7f16e173e640 1 Processor -- start 2024-03-23T14:36:36.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.174+0000 7f16e173e640 1 -- start start 2024-03-23T14:36:36.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16e173e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f16dc0d2b40 0x7f16dc0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:36.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16e173e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16dc0d0460 con 0x7f16dc0d1470 2024-03-23T14:36:36.173 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16e173e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16dc0d06b0 con 0x7f16dc15e400 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16e173e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f16dc0d1050 con 0x7f16dc0d2b40 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16da7fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f16dc0d1470 0x7f16dc0d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:45154/0 (socket says 172.21.15.161:45154) 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16da7fc640 1 -- 172.21.15.161:0/2987753747 learned_addr learned my addr 172.21.15.161:0/2987753747 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16db7fe640 1 --2- 172.21.15.161:0/2987753747 >> v2:172.21.15.183:3300/0 conn(0x7f16dc0d2b40 0x7f16dc0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16d9ffb640 1 -- 172.21.15.161:0/2987753747 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3427105960 0 0) 0x7f16dc0d0460 con 0x7f16dc0d1470 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16d9ffb640 1 -- 172.21.15.161:0/2987753747 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16c4003520 con 0x7f16dc0d1470 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16d9ffb640 1 -- 172.21.15.161:0/2987753747 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 313346887 0 0) 0x7f16dc0d06b0 con 0x7f16dc15e400 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.175+0000 7f16d9ffb640 1 -- 172.21.15.161:0/2987753747 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16dc0d0460 con 0x7f16dc15e400 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.176+0000 7f16db7fe640 1 -- 172.21.15.161:0/2987753747 >> v1:172.21.15.183:6789/0 conn(0x7f16dc15e400 legacy=0x7f16dc0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.176+0000 7f16db7fe640 1 -- 172.21.15.161:0/2987753747 >> v1:172.21.15.161:6789/0 conn(0x7f16dc0d1470 legacy=0x7f16dc0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.176+0000 7f16db7fe640 1 -- 172.21.15.161:0/2987753747 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16dc0d0990 con 0x7f16dc0d2b40 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.176+0000 7f16db7fe640 1 --2- 172.21.15.161:0/2987753747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d2b40 0x7f16dc0b0a80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f16d40097a0 tx=0x7f16d402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfda4fcbabf346f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:36.174 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.176+0000 7f16d9ffb640 1 -- 172.21.15.161:0/2987753747 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4030610 con 0x7f16dc0d2b40 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16d9ffb640 1 -- 172.21.15.161:0/2987753747 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d4030770 con 0x7f16dc0d2b40 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16d9ffb640 1 -- 172.21.15.161:0/2987753747 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d4030b00 con 0x7f16dc0d2b40 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16e173e640 1 -- 172.21.15.161:0/2987753747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d2b40 msgr2=0x7f16dc0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16e173e640 1 --2- 172.21.15.161:0/2987753747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d2b40 0x7f16dc0b0a80 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f16d40097a0 tx=0x7f16d402b960 comp rx=0 tx=0).stop 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16e173e640 1 -- 172.21.15.161:0/2987753747 shutdown_connections 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16e173e640 1 --2- 172.21.15.161:0/2987753747 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d2b40 0x7f16dc0b0a80 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16e173e640 1 -- 172.21.15.161:0/2987753747 >> 172.21.15.161:0/2987753747 conn(0x7f16dc09d190 msgr2=0x7f16dc0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16e173e640 1 -- 172.21.15.161:0/2987753747 shutdown_connections 2024-03-23T14:36:36.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.177+0000 7f16e173e640 1 -- 172.21.15.161:0/2987753747 wait complete. 2024-03-23T14:36:36.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.178+0000 7f16e173e640 1 Processor -- start 2024-03-23T14:36:36.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.178+0000 7f16e173e640 1 -- start start 2024-03-23T14:36:36.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16e173e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d1470 0x7f16dc0b74a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:36.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16e173e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16dc0d2b40 0x7f16dc1593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:36.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16e173e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16dc15e400 0x7f16dc15ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:36.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16e173e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f16dc0caf20 con 0x7f16dc0d2b40 2024-03-23T14:36:36.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16e173e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f16dc0b6a50 con 0x7f16dc0d1470 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16e173e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f16dc0b7010 con 0x7f16dc15e400 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16da7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16dc0d2b40 0x7f16dc1593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16da7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16dc0d2b40 0x7f16dc1593b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:50766/0 (socket says 172.21.15.161:50766) 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16da7fc640 1 -- 172.21.15.161:0/1280995939 learned_addr learned my addr 172.21.15.161:0/1280995939 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16daffd640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d1470 0x7f16dc0b74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.179+0000 7f16db7fe640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16dc15e400 0x7f16dc15ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16da7fc640 1 -- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16dc15e400 msgr2=0x7f16dc15ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16da7fc640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16dc15e400 0x7f16dc15ff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.180 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16da7fc640 1 -- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d1470 msgr2=0x7f16dc0b74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16da7fc640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d1470 0x7f16dc0b74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16da7fc640 1 -- 172.21.15.161:0/1280995939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d4009400 con 0x7f16dc0d2b40 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16daffd640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d1470 0x7f16dc0b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16da7fc640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16dc0d2b40 0x7f16dc1593b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f16d000c440 tx=0x7f16d000c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16c37fe640 1 -- 172.21.15.161:0/1280995939 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0005ce0 con 0x7f16dc0d2b40 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16c37fe640 1 -- 172.21.15.161:0/1280995939 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16d0005e40 con 0x7f16dc0d2b40 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16dc1598f0 con 0x7f16dc0d2b40 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16c37fe640 1 -- 172.21.15.161:0/1280995939 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0007270 con 0x7f16dc0d2b40 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.180+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16dc160aa0 con 0x7f16dc0d2b40 2024-03-23T14:36:36.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.181+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16dc0a1e00 con 0x7f16dc0d2b40 2024-03-23T14:36:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.181+0000 7f16c37fe640 1 -- 172.21.15.161:0/1280995939 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f16d00073d0 con 0x7f16dc0d2b40 2024-03-23T14:36:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.182+0000 7f16c37fe640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f16c806b840 0x7f16c806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.182+0000 7f16c37fe640 1 -- 172.21.15.161:0/1280995939 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f16d0089670 con 0x7f16dc0d2b40 2024-03-23T14:36:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.182+0000 7f16daffd640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f16c806b840 0x7f16c806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.182+0000 7f16daffd640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f16c806b840 0x7f16c806dd00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00f290 tx=0x7f16cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.186+0000 7f16c37fe640 1 -- 172.21.15.161:0/1280995939 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f16d0054100 con 0x7f16dc0d2b40 2024-03-23T14:36:36.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.371+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f16dc0b7150 con 0x7f16dc0d2b40 2024-03-23T14:36:36.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.372+0000 7f16c37fe640 1 -- 172.21.15.161:0/1280995939 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v36) ==== 80+0+0 (secure 0 0 0) 0x7f16d00577c0 con 0x7f16dc0d2b40 2024-03-23T14:36:36.368 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:36:36.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f16c806b840 msgr2=0x7f16c806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f16c806b840 0x7f16c806dd00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f16cc00f290 tx=0x7f16cc009040 comp rx=0 tx=0).stop 2024-03-23T14:36:36.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16dc0d2b40 msgr2=0x7f16dc1593b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:36.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16dc0d2b40 0x7f16dc1593b0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f16d000c440 tx=0x7f16d000c900 comp rx=0 tx=0).stop 2024-03-23T14:36:36.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 shutdown_connections 2024-03-23T14:36:36.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f16c806b840 0x7f16c806dd00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f16dc15e400 0x7f16dc15ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f16dc0d2b40 0x7f16dc1593b0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 --2- 172.21.15.161:0/1280995939 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f16dc0d1470 0x7f16dc0b74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:36.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 >> 172.21.15.161:0/1280995939 conn(0x7f16dc09d190 msgr2=0x7f16dc0b1fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:36.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 shutdown_connections 2024-03-23T14:36:36.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:36.374+0000 7f16e173e640 1 -- 172.21.15.161:0/1280995939 wait complete. 2024-03-23T14:36:36.380 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-23T14:36:36.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:36.381 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T14:36:36.406 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[53885]: fuse finished with error 0 and tester_r 0 2024-03-23T14:36:36.408 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:36:37.511 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:36:37.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:36:42.510 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:36:42.510 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:42.511 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T14:36:42.530 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-23T14:36:42.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.691+0000 7fee7adb6640 1 Processor -- start 2024-03-23T14:36:42.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.691+0000 7fee7adb6640 1 -- start start 2024-03-23T14:36:42.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee7adb6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fee7406cd20 0x7fee7415e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:42.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee7adb6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee74070ee0 con 0x7fee7415a2b0 2024-03-23T14:36:42.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee7adb6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee740b33b0 con 0x7fee74071370 2024-03-23T14:36:42.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee7adb6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fee740b3650 con 0x7fee7406cd20 2024-03-23T14:36:42.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee78b52640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fee7415a2b0 0x7fee7415df80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35176/0 (socket says 172.21.15.161:35176) 2024-03-23T14:36:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee78b52640 1 -- 172.21.15.161:0/3423488401 learned_addr learned my addr 172.21.15.161:0/3423488401 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee6bfff640 1 --2- 172.21.15.161:0/3423488401 >> v2:172.21.15.183:3300/0 conn(0x7fee7406cd20 0x7fee7415e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee6b7fe640 1 -- 172.21.15.161:0/3423488401 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2149215383 0 0) 0x7fee74070ee0 con 0x7fee7415a2b0 2024-03-23T14:36:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee6b7fe640 1 -- 172.21.15.161:0/3423488401 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee5c003520 con 0x7fee7415a2b0 2024-03-23T14:36:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee6bfff640 1 -- 172.21.15.161:0/3423488401 >> v1:172.21.15.183:6789/0 conn(0x7fee74071370 legacy=0x7fee740717e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee6bfff640 1 -- 172.21.15.161:0/3423488401 >> v1:172.21.15.161:6789/0 conn(0x7fee7415a2b0 legacy=0x7fee7415df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:42.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.692+0000 7fee6bfff640 1 -- 172.21.15.161:0/3423488401 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee740b3930 con 0x7fee7406cd20 2024-03-23T14:36:42.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.693+0000 7fee6bfff640 1 --2- 172.21.15.161:0/3423488401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 0x7fee7415e690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fee6400a940 tx=0x7fee6402ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3ef5d125fd4d924 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:42.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.693+0000 7fee6b7fe640 1 -- 172.21.15.161:0/3423488401 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6402dbe0 con 0x7fee7406cd20 2024-03-23T14:36:42.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.693+0000 7fee6b7fe640 1 -- 172.21.15.161:0/3423488401 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee6402dd40 con 0x7fee7406cd20 2024-03-23T14:36:42.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.693+0000 7fee6b7fe640 1 -- 172.21.15.161:0/3423488401 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee64035660 con 0x7fee7406cd20 2024-03-23T14:36:42.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.694+0000 7fee7adb6640 1 -- 172.21.15.161:0/3423488401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 msgr2=0x7fee7415e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:42.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.694+0000 7fee7adb6640 1 --2- 172.21.15.161:0/3423488401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 0x7fee7415e690 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fee6400a940 tx=0x7fee6402ce30 comp rx=0 tx=0).stop 2024-03-23T14:36:42.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.694+0000 7fee7adb6640 1 -- 172.21.15.161:0/3423488401 shutdown_connections 2024-03-23T14:36:42.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.694+0000 7fee7adb6640 1 --2- 172.21.15.161:0/3423488401 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 0x7fee7415e690 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:42.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.694+0000 7fee7adb6640 1 -- 172.21.15.161:0/3423488401 >> 172.21.15.161:0/3423488401 conn(0x7fee7406b260 msgr2=0x7fee7406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:42.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.694+0000 7fee7adb6640 1 -- 172.21.15.161:0/3423488401 shutdown_connections 2024-03-23T14:36:42.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.694+0000 7fee7adb6640 1 -- 172.21.15.161:0/3423488401 wait complete. 2024-03-23T14:36:42.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.695+0000 7fee7adb6640 1 Processor -- start 2024-03-23T14:36:42.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.695+0000 7fee7adb6640 1 -- start start 2024-03-23T14:36:42.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee7adb6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 0x7fee74146c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee7adb6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee74071370 0x7fee740a9e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee7adb6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee7415a2b0 0x7fee740cc8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee7adb6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fee74147920 con 0x7fee74071370 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee7adb6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fee7414a850 con 0x7fee7406cd20 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee7adb6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fee7414ab30 con 0x7fee7415a2b0 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee74071370 0x7fee740a9e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee78b52640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 0x7fee74146c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee6bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee74071370 0x7fee740a9e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41942/0 (socket says 172.21.15.161:41942) 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee6bfff640 1 -- 172.21.15.161:0/2151237766 learned_addr learned my addr 172.21.15.161:0/2151237766 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee79353640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee7415a2b0 0x7fee740cc8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee6bfff640 1 -- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee7415a2b0 msgr2=0x7fee740cc8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:42.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee6bfff640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee7415a2b0 0x7fee740cc8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee6bfff640 1 -- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 msgr2=0x7fee74146c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee6bfff640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 0x7fee74146c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee6bfff640 1 -- 172.21.15.161:0/2151237766 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee6400a5a0 con 0x7fee74071370 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.696+0000 7fee78b52640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 0x7fee74146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.697+0000 7fee6bfff640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee74071370 0x7fee740a9e50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fee64038040 tx=0x7fee640046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.697+0000 7fee697fa640 1 -- 172.21.15.161:0/2151237766 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6403e610 con 0x7fee74071370 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.697+0000 7fee697fa640 1 -- 172.21.15.161:0/2151237766 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee6403e7b0 con 0x7fee74071370 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.697+0000 7fee697fa640 1 -- 172.21.15.161:0/2151237766 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee6403eb60 con 0x7fee74071370 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.697+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee7414ae10 con 0x7fee74071370 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.697+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee7414b3d0 con 0x7fee74071370 2024-03-23T14:36:42.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.698+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee7414b5c0 con 0x7fee74071370 2024-03-23T14:36:42.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.698+0000 7fee697fa640 1 -- 172.21.15.161:0/2151237766 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee6403c040 con 0x7fee74071370 2024-03-23T14:36:42.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.699+0000 7fee697fa640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fee6006b810 0x7fee6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:42.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.699+0000 7fee697fa640 1 -- 172.21.15.161:0/2151237766 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fee640ac480 con 0x7fee74071370 2024-03-23T14:36:42.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.699+0000 7fee78b52640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fee6006b810 0x7fee6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:42.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.699+0000 7fee78b52640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fee6006b810 0x7fee6006dcd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee700011e0 tx=0x7fee70007d30 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:42.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.703+0000 7fee697fa640 1 -- 172.21.15.161:0/2151237766 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee64076e30 con 0x7fee74071370 2024-03-23T14:36:42.933 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.938+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7fee74068160 con 0x7fee74071370 2024-03-23T14:36:42.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.940+0000 7fee697fa640 1 -- 172.21.15.161:0/2151237766 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v14) ==== 184+0+141 (secure 0 0 0) 0x7fee6407a4f0 con 0x7fee74071370 2024-03-23T14:36:42.936 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:36:42.936 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:36:42.936 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-23T14:36:42.936 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-23T14:36:42.936 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-23T14:36:42.936 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fee6006b810 msgr2=0x7fee6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fee6006b810 0x7fee6006dcd0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fee700011e0 tx=0x7fee70007d30 comp rx=0 tx=0).stop 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee74071370 msgr2=0x7fee740a9e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee74071370 0x7fee740a9e50 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7fee64038040 tx=0x7fee640046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 shutdown_connections 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fee6006b810 0x7fee6006dcd0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee7415a2b0 0x7fee740cc8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee74071370 0x7fee740a9e50 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:42.938 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 --2- 172.21.15.161:0/2151237766 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee7406cd20 0x7fee74146c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:42.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 >> 172.21.15.161:0/2151237766 conn(0x7fee7406b260 msgr2=0x7fee7414d170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:42.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 shutdown_connections 2024-03-23T14:36:42.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:42.942+0000 7fee7adb6640 1 -- 172.21.15.161:0/2151237766 wait complete. 2024-03-23T14:36:42.948 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T14:36:42.949 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T14:36:42.991 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T14:36:43.039 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T14:36:43.039 INFO:tasks.cephfs.mount:Terminating background process 2024-03-23T14:36:43.039 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:43.039 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:43.054 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18767 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T14:36:43.054 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:36:43.055 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:43.055 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T14:36:43.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:36:43.109 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-03-23T14:36:43.110 INFO:tasks.cephfs.mount:Terminating background process 2024-03-23T14:36:43.110 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:36:43.110 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:43.126 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056359,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17751 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T14:36:43.126 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:36:43.126 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:43.126 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T14:36:43.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:36:43.181 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T14:36:43.181 INFO:tasks.cephfs.mount:Terminating background process 2024-03-23T14:36:43.181 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:43.181 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:43.197 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18767 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T14:36:43.197 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:36:43.197 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:43.197 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T14:36:43.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:36:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T14:36:43.252 INFO:tasks.cephfs.mount:Terminating background process 2024-03-23T14:36:43.252 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:36:43.252 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:43.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056359,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17751 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T14:36:43.268 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:36:43.268 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:36:43.268 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T14:36:43.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:36:43.322 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T14:36:43.323 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:36:43.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.427+0000 7fee158bb640 1 Processor -- start 2024-03-23T14:36:43.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.427+0000 7fee158bb640 1 -- start start 2024-03-23T14:36:43.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee158bb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fee100b4260 0x7fee1015b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee158bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee100b3610 0x7fee100b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee158bb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee10070b90 con 0x7fee1006eb30 2024-03-23T14:36:43.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee158bb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fee100b3fe0 con 0x7fee100b4260 2024-03-23T14:36:43.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee158bb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fee1015ab90 con 0x7fee100b3610 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0e7fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fee100b4260 0x7fee1015b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fee1006eb30 0x7fee100b4870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35192/0 (socket says 172.21.15.161:35192) 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0effd640 1 -- 172.21.15.161:0/1343744446 learned_addr learned my addr 172.21.15.161:0/1343744446 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0e7fc640 1 --2- 172.21.15.161:0/1343744446 >> v2:172.21.15.161:3300/0 conn(0x7fee100b4260 0x7fee1015b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41952/0 (socket says 172.21.15.161:41952) 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0f7fe640 1 --2- 172.21.15.161:0/1343744446 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee100b3610 0x7fee100b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0e7fc640 1 -- 172.21.15.161:0/1343744446 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee100b3610 msgr2=0x7fee100b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0e7fc640 1 --2- 172.21.15.161:0/1343744446 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee100b3610 0x7fee100b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0e7fc640 1 -- 172.21.15.161:0/1343744446 >> v1:172.21.15.161:6789/0 conn(0x7fee1006eb30 legacy=0x7fee100b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0dffb640 1 -- 172.21.15.161:0/1343744446 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1066452046 0 0) 0x7fee10070b90 con 0x7fee1006eb30 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.428+0000 7fee0e7fc640 1 -- 172.21.15.161:0/1343744446 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee1015b0d0 con 0x7fee100b4260 2024-03-23T14:36:43.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.429+0000 7fee0e7fc640 1 --2- 172.21.15.161:0/1343744446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b4260 0x7fee1015b560 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fee0400a940 tx=0x7fee0402cd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7db9a2d0d5fbcc9f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:43.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.429+0000 7fee0dffb640 1 -- 172.21.15.161:0/1343744446 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee0402db30 con 0x7fee100b4260 2024-03-23T14:36:43.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.429+0000 7fee0dffb640 1 -- 172.21.15.161:0/1343744446 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee0402dc90 con 0x7fee100b4260 2024-03-23T14:36:43.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.429+0000 7fee0dffb640 1 -- 172.21.15.161:0/1343744446 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04035440 con 0x7fee100b4260 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.430+0000 7fee158bb640 1 -- 172.21.15.161:0/1343744446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b4260 msgr2=0x7fee1015b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.430+0000 7fee158bb640 1 --2- 172.21.15.161:0/1343744446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b4260 0x7fee1015b560 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fee0400a940 tx=0x7fee0402cd80 comp rx=0 tx=0).stop 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.430+0000 7fee158bb640 1 -- 172.21.15.161:0/1343744446 shutdown_connections 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.430+0000 7fee158bb640 1 --2- 172.21.15.161:0/1343744446 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee100b3610 0x7fee100b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.430+0000 7fee158bb640 1 --2- 172.21.15.161:0/1343744446 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b4260 0x7fee1015b560 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.430+0000 7fee158bb640 1 -- 172.21.15.161:0/1343744446 >> 172.21.15.161:0/1343744446 conn(0x7fee100b17f0 msgr2=0x7fee100b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.430+0000 7fee158bb640 1 -- 172.21.15.161:0/1343744446 shutdown_connections 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.430+0000 7fee158bb640 1 -- 172.21.15.161:0/1343744446 wait complete. 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.431+0000 7fee158bb640 1 Processor -- start 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.431+0000 7fee158bb640 1 -- start start 2024-03-23T14:36:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee158bb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee1006eb30 0x7fee1014a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee158bb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b3610 0x7fee10159360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee158bb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee100b4260 0x7fee101598a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee158bb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fee101f4270 con 0x7fee100b3610 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee158bb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fee101f4550 con 0x7fee100b4260 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee158bb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fee101f4830 con 0x7fee1006eb30 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b3610 0x7fee10159360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b3610 0x7fee10159360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41966/0 (socket says 172.21.15.161:41966) 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 -- 172.21.15.161:0/3342355983 learned_addr learned my addr 172.21.15.161:0/3342355983 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0effd640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee1006eb30 0x7fee1014a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0f7fe640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee100b4260 0x7fee101598a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 -- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee1006eb30 msgr2=0x7fee1014a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee1006eb30 0x7fee1014a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 -- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee100b4260 msgr2=0x7fee101598a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee100b4260 0x7fee101598a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 -- 172.21.15.161:0/3342355983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee0400a5a0 con 0x7fee100b3610 2024-03-23T14:36:43.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.432+0000 7fee0e7fc640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b3610 0x7fee10159360 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fee0400a940 tx=0x7fee040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:43.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.433+0000 7fedf7fff640 1 -- 172.21.15.161:0/3342355983 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04004030 con 0x7fee100b3610 2024-03-23T14:36:43.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.433+0000 7fedf7fff640 1 -- 172.21.15.161:0/3342355983 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fee04004190 con 0x7fee100b3610 2024-03-23T14:36:43.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.433+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee101f4b10 con 0x7fee100b3610 2024-03-23T14:36:43.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.433+0000 7fedf7fff640 1 -- 172.21.15.161:0/3342355983 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee04003610 con 0x7fee100b3610 2024-03-23T14:36:43.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.433+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee1014ab10 con 0x7fee100b3610 2024-03-23T14:36:43.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.434+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee1004f550 con 0x7fee100b3610 2024-03-23T14:36:43.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.434+0000 7fedf7fff640 1 -- 172.21.15.161:0/3342355983 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fee0403c040 con 0x7fee100b3610 2024-03-23T14:36:43.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.435+0000 7fedf7fff640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fedfc06b810 0x7fedfc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.435+0000 7fedf7fff640 1 -- 172.21.15.161:0/3342355983 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fee040ac720 con 0x7fee100b3610 2024-03-23T14:36:43.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.435+0000 7fee0effd640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fedfc06b810 0x7fedfc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.435+0000 7fee0effd640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fedfc06b810 0x7fedfc06dcd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fee000012e0 tx=0x7fee00001700 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:43.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.439+0000 7fedf7fff640 1 -- 172.21.15.161:0/3342355983 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fee04077180 con 0x7fee100b3610 2024-03-23T14:36:43.641 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.646+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fee101f43b0 con 0x7fee100b3610 2024-03-23T14:36:43.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.646+0000 7fedf7fff640 1 -- 172.21.15.161:0/3342355983 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5182 (secure 0 0 0) 0x7fee101f43b0 con 0x7fee100b3610 2024-03-23T14:36:43.642 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:43.642 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/626265302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":626265302},{"type":"v1","addr":"172.21.15.183:6837","nonce":626265302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5220,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/208344896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":208344896},{"type":"v1","addr":"172.21.15.161:6833","nonce":208344896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/82303731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":82303731},{"type":"v1","addr":"172.21.15.161:6835","nonce":82303731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:06.267736+0000","modified":"2024-03-23T14:36:19.582504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5235":{"gid":5235,"name":"d","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6836/2372368263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2372368263},{"type":"v1","addr":"172.21.15.183:6836","nonce":2372368263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5235,"qdb_cluster":[5235]},"id":2}]} 2024-03-23T14:36:43.643 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 16 2024-03-23T14:36:43.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fedfc06b810 msgr2=0x7fedfc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fedfc06b810 0x7fedfc06dcd0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fee000012e0 tx=0x7fee00001700 comp rx=0 tx=0).stop 2024-03-23T14:36:43.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b3610 msgr2=0x7fee10159360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b3610 0x7fee10159360 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fee0400a940 tx=0x7fee040046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 shutdown_connections 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fedfc06b810 0x7fedfc06dcd0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fee100b4260 0x7fee101598a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fee100b3610 0x7fee10159360 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 --2- 172.21.15.161:0/3342355983 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fee1006eb30 0x7fee1014a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 >> 172.21.15.161:0/3342355983 conn(0x7fee100b17f0 msgr2=0x7fee10154460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 shutdown_connections 2024-03-23T14:36:43.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.648+0000 7fee158bb640 1 -- 172.21.15.161:0/3342355983 wait complete. 2024-03-23T14:36:43.655 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:43.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.761+0000 7fddac62e640 1 Processor -- start 2024-03-23T14:36:43.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.761+0000 7fddac62e640 1 -- start start 2024-03-23T14:36:43.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fddac62e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdda4072100 0x7fdda415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fddac62e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdda40724e0 0x7fdda406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fddac62e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdda415be70 con 0x7fdda415b930 2024-03-23T14:36:43.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fddac62e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fdda415c430 con 0x7fdda40724e0 2024-03-23T14:36:43.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fddac62e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fdda415c150 con 0x7fdda4072100 2024-03-23T14:36:43.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fdda9bc9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdda40724e0 0x7fdda406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fdda9bc9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fdda40724e0 0x7fdda406e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41974/0 (socket says 172.21.15.161:41974) 2024-03-23T14:36:43.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fdda9bc9640 1 -- 172.21.15.161:0/4042575435 learned_addr learned my addr 172.21.15.161:0/4042575435 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:43.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fddaa3ca640 1 --2- 172.21.15.161:0/4042575435 >> v2:172.21.15.183:3300/0 conn(0x7fdda4072100 0x7fdda415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fdda9bc9640 1 -- 172.21.15.161:0/4042575435 >> v2:172.21.15.183:3300/0 conn(0x7fdda4072100 msgr2=0x7fdda415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:43.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fdda9bc9640 1 --2- 172.21.15.161:0/4042575435 >> v2:172.21.15.183:3300/0 conn(0x7fdda4072100 0x7fdda415b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fdda9bc9640 1 -- 172.21.15.161:0/4042575435 >> v1:172.21.15.183:6789/0 conn(0x7fdda415b930 legacy=0x7fdda415e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.762+0000 7fdda9bc9640 1 -- 172.21.15.161:0/4042575435 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda40b2c70 con 0x7fdda40724e0 2024-03-23T14:36:43.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.763+0000 7fdda9bc9640 1 --2- 172.21.15.161:0/4042575435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda406e590 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdda406cac0 tx=0x7fdd9402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21979c50cad10686 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:43.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.763+0000 7fdda93c8640 1 -- 172.21.15.161:0/4042575435 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94030860 con 0x7fdda40724e0 2024-03-23T14:36:43.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.763+0000 7fdda93c8640 1 -- 172.21.15.161:0/4042575435 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd940309c0 con 0x7fdda40724e0 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.763+0000 7fdda93c8640 1 -- 172.21.15.161:0/4042575435 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94030d50 con 0x7fdda40724e0 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.763+0000 7fddac62e640 1 -- 172.21.15.161:0/4042575435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 msgr2=0x7fdda406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.763+0000 7fddac62e640 1 --2- 172.21.15.161:0/4042575435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda406e590 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fdda406cac0 tx=0x7fdd9402b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.764+0000 7fddac62e640 1 -- 172.21.15.161:0/4042575435 shutdown_connections 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.764+0000 7fddac62e640 1 --2- 172.21.15.161:0/4042575435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda406e590 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.764+0000 7fddac62e640 1 --2- 172.21.15.161:0/4042575435 >> v2:172.21.15.183:3300/0 conn(0x7fdda4072100 0x7fdda415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.764+0000 7fddac62e640 1 -- 172.21.15.161:0/4042575435 >> 172.21.15.161:0/4042575435 conn(0x7fdda40b17f0 msgr2=0x7fdda40b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.764+0000 7fddac62e640 1 -- 172.21.15.161:0/4042575435 shutdown_connections 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.764+0000 7fddac62e640 1 -- 172.21.15.161:0/4042575435 wait complete. 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fddac62e640 1 Processor -- start 2024-03-23T14:36:43.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fddac62e640 1 -- start start 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fddac62e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdda4072100 0x7fdda414f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fddac62e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda40ce580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fddac62e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdda415b930 0x7fdda41f6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fddac62e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdda40ceac0 con 0x7fdda40724e0 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fddac62e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdda414f6e0 con 0x7fdda4072100 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fddac62e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fdda414f9c0 con 0x7fdda415b930 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.765+0000 7fdda9bc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda40ce580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fddaa3ca640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdda4072100 0x7fdda414f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdda9bc9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda40ce580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41978/0 (socket says 172.21.15.161:41978) 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fddaabcb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdda415b930 0x7fdda41f6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdda9bc9640 1 -- 172.21.15.161:0/3596624193 learned_addr learned my addr 172.21.15.161:0/3596624193 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdda9bc9640 1 -- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdda415b930 msgr2=0x7fdda41f6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdda9bc9640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdda415b930 0x7fdda41f6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdda9bc9640 1 -- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdda4072100 msgr2=0x7fdda414f090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdda9bc9640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdda4072100 0x7fdda414f090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdda9bc9640 1 -- 172.21.15.161:0/3596624193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd940094d0 con 0x7fdda40724e0 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdda9bc9640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda40ce580 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdd9402b9e0 tx=0x7fdd94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdd9affd640 1 -- 172.21.15.161:0/3596624193 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94039c90 con 0x7fdda40724e0 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdd9affd640 1 -- 172.21.15.161:0/3596624193 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd94039e10 con 0x7fdda40724e0 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda414fca0 con 0x7fdda40724e0 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fdd9affd640 1 -- 172.21.15.161:0/3596624193 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd94043640 con 0x7fdda40724e0 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.766+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda4150260 con 0x7fdda40724e0 2024-03-23T14:36:43.765 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.767+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda40735e0 con 0x7fdda40724e0 2024-03-23T14:36:43.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.767+0000 7fdd9affd640 1 -- 172.21.15.161:0/3596624193 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd940437a0 con 0x7fdda40724e0 2024-03-23T14:36:43.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.768+0000 7fdd9affd640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd8806b840 0x7fdd8806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:43.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.768+0000 7fdd9affd640 1 -- 172.21.15.161:0/3596624193 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fdd9403e070 con 0x7fdda40724e0 2024-03-23T14:36:43.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.768+0000 7fddaa3ca640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd8806b840 0x7fdd8806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:43.766 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.768+0000 7fddaa3ca640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd8806b840 0x7fdd8806dd00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009b40 tx=0x7fdda0009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:43.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.772+0000 7fdd9affd640 1 -- 172.21.15.161:0/3596624193 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd94077c40 con 0x7fdda40724e0 2024-03-23T14:36:43.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.944+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdda406ccd0 con 0x7fdda40724e0 2024-03-23T14:36:43.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.945+0000 7fdd9affd640 1 -- 172.21.15.161:0/3596624193 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7fdd9407b300 con 0x7fdda40724e0 2024-03-23T14:36:43.941 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:43.941 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd8806b840 msgr2=0x7fdd8806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd8806b840 0x7fdd8806dd00 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009b40 tx=0x7fdda0009190 comp rx=0 tx=0).stop 2024-03-23T14:36:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 msgr2=0x7fdda40ce580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:43.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda40ce580 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fdd9402b9e0 tx=0x7fdd94004730 comp rx=0 tx=0).stop 2024-03-23T14:36:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 shutdown_connections 2024-03-23T14:36:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd8806b840 0x7fdd8806dd00 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdda415b930 0x7fdda41f6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdda40724e0 0x7fdda40ce580 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 --2- 172.21.15.161:0/3596624193 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdda4072100 0x7fdda414f090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 >> 172.21.15.161:0/3596624193 conn(0x7fdda40b17f0 msgr2=0x7fdda40b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 shutdown_connections 2024-03-23T14:36:43.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:43.947+0000 7fddac62e640 1 -- 172.21.15.161:0/3596624193 wait complete. 2024-03-23T14:36:43.954 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:04.688180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:05.241774+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.3799999952316284, 'score_stable': 1.3799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'data_new', 'create_time': '2024-03-23T14:36:18.738957+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '36', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:43.955 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:44.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.065+0000 7f138ef5e640 1 Processor -- start 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.065+0000 7f138ef5e640 1 -- start start 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.065+0000 7f138ef5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1388161820 0x7f13880d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.065+0000 7f138ef5e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f13880d1270 0x7f13880d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.065+0000 7f138ef5e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1388161c00 con 0x7f13880d29d0 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.065+0000 7f138ef5e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f13880d2e20 con 0x7f13880d1270 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.065+0000 7f138ef5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f13880cea30 con 0x7f1388161820 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f1387fff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f13880d1270 0x7f13880d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f138ccfa640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1388161820 0x7f13880d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f138d4fb640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f13880d29d0 0x7f1388161e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:33998/0 (socket says 172.21.15.161:33998) 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f138d4fb640 1 -- 172.21.15.161:0/3838249336 learned_addr learned my addr 172.21.15.161:0/3838249336 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f1387fff640 1 -- 172.21.15.161:0/3838249336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1388161820 msgr2=0x7f13880d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:44.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f1387fff640 1 --2- 172.21.15.161:0/3838249336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1388161820 0x7f13880d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f1387fff640 1 -- 172.21.15.161:0/3838249336 >> v1:172.21.15.183:6789/0 conn(0x7f13880d29d0 legacy=0x7f1388161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f1387fff640 1 -- 172.21.15.161:0/3838249336 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13880cfd70 con 0x7f13880d1270 2024-03-23T14:36:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.066+0000 7f138ccfa640 1 --2- 172.21.15.161:0/3838249336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1388161820 0x7f13880d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f1387fff640 1 --2- 172.21.15.161:0/3838249336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13880d1270 0x7f13880d2490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f13740097a0 tx=0x7f137402b710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eabed0027f9d2b26 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f13877fe640 1 -- 172.21.15.161:0/3838249336 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374030710 con 0x7f13880d1270 2024-03-23T14:36:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f13877fe640 1 -- 172.21.15.161:0/3838249336 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137402e050 con 0x7f13880d1270 2024-03-23T14:36:44.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f13877fe640 1 -- 172.21.15.161:0/3838249336 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1374030a90 con 0x7f13880d1270 2024-03-23T14:36:44.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f138ef5e640 1 -- 172.21.15.161:0/3838249336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13880d1270 msgr2=0x7f13880d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f138ef5e640 1 --2- 172.21.15.161:0/3838249336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13880d1270 0x7f13880d2490 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f13740097a0 tx=0x7f137402b710 comp rx=0 tx=0).stop 2024-03-23T14:36:44.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f138ef5e640 1 -- 172.21.15.161:0/3838249336 shutdown_connections 2024-03-23T14:36:44.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f138ef5e640 1 --2- 172.21.15.161:0/3838249336 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13880d1270 0x7f13880d2490 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f138ef5e640 1 --2- 172.21.15.161:0/3838249336 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1388161820 0x7f13880d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f138ef5e640 1 -- 172.21.15.161:0/3838249336 >> 172.21.15.161:0/3838249336 conn(0x7f138809d190 msgr2=0x7f13880d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f138ef5e640 1 -- 172.21.15.161:0/3838249336 shutdown_connections 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.067+0000 7f138ef5e640 1 -- 172.21.15.161:0/3838249336 wait complete. 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.068+0000 7f138ef5e640 1 Processor -- start 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ef5e640 1 -- start start 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ef5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13880d1270 0x7f13881f46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ef5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13880d29d0 0x7f138814f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ef5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1388161820 0x7f13881f7210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ef5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f13881f4230 con 0x7f13880d1270 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ef5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f13881f4c00 con 0x7f1388161820 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ef5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f138814f840 con 0x7f13880d29d0 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ccfa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13880d1270 0x7f13881f46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.069+0000 7f138ccfa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13880d1270 0x7f13881f46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41980/0 (socket says 172.21.15.161:41980) 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ccfa640 1 -- 172.21.15.161:0/222194105 learned_addr learned my addr 172.21.15.161:0/222194105 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f1387fff640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13880d29d0 0x7f138814f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138d4fb640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1388161820 0x7f13881f7210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ccfa640 1 -- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13880d29d0 msgr2=0x7f138814f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ccfa640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13880d29d0 0x7f138814f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ccfa640 1 -- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1388161820 msgr2=0x7f13881f7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ccfa640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1388161820 0x7f13881f7210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ccfa640 1 -- 172.21.15.161:0/222194105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f137c009400 con 0x7f13880d1270 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f1387fff640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13880d29d0 0x7f138814f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ccfa640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13880d1270 0x7f13881f46c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f137c00ebe0 tx=0x7f137c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f13857fa640 1 -- 172.21.15.161:0/222194105 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c009760 con 0x7f13880d1270 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f13857fa640 1 -- 172.21.15.161:0/222194105 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f137c00cc30 con 0x7f13880d1270 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1374009400 con 0x7f13880d1270 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f13857fa640 1 -- 172.21.15.161:0/222194105 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f137c00eee0 con 0x7f13880d1270 2024-03-23T14:36:44.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.070+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13881f7cd0 con 0x7f13880d1270 2024-03-23T14:36:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.071+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f138804f4e0 con 0x7f13880d1270 2024-03-23T14:36:44.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.072+0000 7f13857fa640 1 -- 172.21.15.161:0/222194105 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f137c013470 con 0x7f13880d1270 2024-03-23T14:36:44.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.072+0000 7f13857fa640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f136c06b810 0x7f136c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.072+0000 7f13857fa640 1 -- 172.21.15.161:0/222194105 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f137c08a570 con 0x7f13880d1270 2024-03-23T14:36:44.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.072+0000 7f1387fff640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f136c06b810 0x7f136c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.073+0000 7f1387fff640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f136c06b810 0x7f136c06dcd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1374034000 tx=0x7f1374008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.072 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.076+0000 7f13857fa640 1 -- 172.21.15.161:0/222194105 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f137c054f40 con 0x7f13880d1270 2024-03-23T14:36:44.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.249+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f13880d2270 con 0x7f13880d1270 2024-03-23T14:36:44.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.250+0000 7f13857fa640 1 -- 172.21.15.161:0/222194105 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v36) ==== 93+0+10 (secure 0 0 0) 0x7f137c058600 con 0x7f13880d1270 2024-03-23T14:36:44.246 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f136c06b810 msgr2=0x7f136c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f136c06b810 0x7f136c06dcd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1374034000 tx=0x7f1374008030 comp rx=0 tx=0).stop 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13880d1270 msgr2=0x7f13881f46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13880d1270 0x7f13881f46c0 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7f137c00ebe0 tx=0x7f137c00c520 comp rx=0 tx=0).stop 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 shutdown_connections 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f136c06b810 0x7f136c06dcd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1388161820 0x7f13881f7210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13880d29d0 0x7f138814f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 --2- 172.21.15.161:0/222194105 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13880d1270 0x7f13881f46c0 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 >> 172.21.15.161:0/222194105 conn(0x7f138809d190 msgr2=0x7f138815e430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 shutdown_connections 2024-03-23T14:36:44.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.252+0000 7f138ef5e640 1 -- 172.21.15.161:0/222194105 wait complete. 2024-03-23T14:36:44.259 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:36:44.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.366+0000 7f97a89d7640 1 Processor -- start 2024-03-23T14:36:44.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.366+0000 7f97a89d7640 1 -- start start 2024-03-23T14:36:44.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a89d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a415c1d0 0x7f97a40d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a89d7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f97a40ce3c0 0x7f97a40d0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a89d7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97a40d0850 0x7f97a40cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a89d7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f97a40d0c80 con 0x7f97a40d0850 2024-03-23T14:36:44.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a89d7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f97a40ceb30 con 0x7f97a40ce3c0 2024-03-23T14:36:44.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a89d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f97a40cee10 con 0x7f97a415c1d0 2024-03-23T14:36:44.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97a40d0850 0x7f97a40cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f979bfff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f97a40ce3c0 0x7f97a40d0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a259c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a415c1d0 0x7f97a40d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f97a40d0850 0x7f97a40cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41994/0 (socket says 172.21.15.161:41994) 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f979bfff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f97a40ce3c0 0x7f97a40d0310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:36048/0 (socket says 172.21.15.161:36048) 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.367+0000 7f97a2d9d640 1 -- 172.21.15.161:0/817009625 learned_addr learned my addr 172.21.15.161:0/817009625 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.368+0000 7f97a2d9d640 1 -- 172.21.15.161:0/817009625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a415c1d0 msgr2=0x7f97a40d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.368+0000 7f97a2d9d640 1 --2- 172.21.15.161:0/817009625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a415c1d0 0x7f97a40d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.368+0000 7f97a2d9d640 1 -- 172.21.15.161:0/817009625 >> v2:172.21.15.183:3300/0 conn(0x7f97a40ce3c0 msgr2=0x7f97a40d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.368+0000 7f97a2d9d640 1 --2- 172.21.15.161:0/817009625 >> v2:172.21.15.183:3300/0 conn(0x7f97a40ce3c0 0x7f97a40d0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.368+0000 7f97a2d9d640 1 -- 172.21.15.161:0/817009625 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97a40cf0f0 con 0x7f97a40d0850 2024-03-23T14:36:44.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.368+0000 7f979bfff640 1 --2- 172.21.15.161:0/817009625 >> v2:172.21.15.183:3300/0 conn(0x7f97a40ce3c0 0x7f97a40d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.368+0000 7f97a2d9d640 1 --2- 172.21.15.161:0/817009625 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a40d0850 0x7f97a40cc8a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f97a4067d30 tx=0x7f979c02be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f2c48000ba627cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a1d9b640 1 -- 172.21.15.161:0/817009625 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c02c9e0 con 0x7f97a40d0850 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a1d9b640 1 -- 172.21.15.161:0/817009625 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f979c02cb40 con 0x7f97a40d0850 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a1d9b640 1 -- 172.21.15.161:0/817009625 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c0343f0 con 0x7f97a40d0850 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 -- 172.21.15.161:0/817009625 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a40d0850 msgr2=0x7f97a40cc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 --2- 172.21.15.161:0/817009625 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a40d0850 0x7f97a40cc8a0 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f97a4067d30 tx=0x7f979c02be40 comp rx=0 tx=0).stop 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 -- 172.21.15.161:0/817009625 shutdown_connections 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 --2- 172.21.15.161:0/817009625 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a40d0850 0x7f97a40cc8a0 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 --2- 172.21.15.161:0/817009625 >> v2:172.21.15.183:3300/0 conn(0x7f97a40ce3c0 0x7f97a40d0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 --2- 172.21.15.161:0/817009625 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a415c1d0 0x7f97a40d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 -- 172.21.15.161:0/817009625 >> 172.21.15.161:0/817009625 conn(0x7f97a409aa40 msgr2=0x7f97a409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 -- 172.21.15.161:0/817009625 shutdown_connections 2024-03-23T14:36:44.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.369+0000 7f97a89d7640 1 -- 172.21.15.161:0/817009625 wait complete. 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.370+0000 7f97a89d7640 1 Processor -- start 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a89d7640 1 -- start start 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a89d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97a40ce3c0 0x7f97a40b63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a89d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a40d0850 0x7f97a41f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a89d7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a415c1d0 0x7f97a40c7d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a89d7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f97a40b5f90 con 0x7f97a415c1d0 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a89d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f97a41f42c0 con 0x7f97a40ce3c0 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a89d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f97a41f45a0 con 0x7f97a40d0850 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a415c1d0 0x7f97a40c7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a415c1d0 0x7f97a40c7d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42000/0 (socket says 172.21.15.161:42000) 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a2d9d640 1 -- 172.21.15.161:0/2435504960 learned_addr learned my addr 172.21.15.161:0/2435504960 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f97a259c640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97a40ce3c0 0x7f97a40b63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.371 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.371+0000 7f979bfff640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a40d0850 0x7f97a41f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a2d9d640 1 -- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a40d0850 msgr2=0x7f97a41f3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a2d9d640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a40d0850 0x7f97a41f3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a2d9d640 1 -- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97a40ce3c0 msgr2=0x7f97a40b63b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a2d9d640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97a40ce3c0 0x7f97a40b63b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a2d9d640 1 -- 172.21.15.161:0/2435504960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f978c009400 con 0x7f97a415c1d0 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a259c640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97a40ce3c0 0x7f97a40b63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a2d9d640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a415c1d0 0x7f97a40c7d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f979c036040 tx=0x7f979c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f979affd640 1 -- 172.21.15.161:0/2435504960 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c007e70 con 0x7f97a415c1d0 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f979affd640 1 -- 172.21.15.161:0/2435504960 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f979c009510 con 0x7f97a415c1d0 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f979c009b30 con 0x7f97a415c1d0 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f979affd640 1 -- 172.21.15.161:0/2435504960 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f979c03e690 con 0x7f97a415c1d0 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.372+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97a41f4880 con 0x7f97a415c1d0 2024-03-23T14:36:44.372 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.373+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97a40b60d0 con 0x7f97a415c1d0 2024-03-23T14:36:44.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.374+0000 7f979affd640 1 -- 172.21.15.161:0/2435504960 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f979c03e7f0 con 0x7f97a415c1d0 2024-03-23T14:36:44.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.374+0000 7f979affd640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f978406b810 0x7f978406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.374+0000 7f979affd640 1 -- 172.21.15.161:0/2435504960 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f979c0accf0 con 0x7f97a415c1d0 2024-03-23T14:36:44.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.374+0000 7f97a259c640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f978406b810 0x7f978406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.373 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.374+0000 7f97a259c640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f978406b810 0x7f978406dcd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9794009e90 tx=0x7f9794009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.374 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.378+0000 7f979affd640 1 -- 172.21.15.161:0/2435504960 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f97a40b60d0 con 0x7f97a415c1d0 2024-03-23T14:36:44.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.551+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f97a40b60d0 con 0x7f97a415c1d0 2024-03-23T14:36:44.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.551+0000 7f979affd640 1 -- 172.21.15.161:0/2435504960 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v36) ==== 104+0+11 (secure 0 0 0) 0x7f97a40b60d0 con 0x7f97a415c1d0 2024-03-23T14:36:44.547 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f978406b810 msgr2=0x7f978406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f978406b810 0x7f978406dcd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f9794009e90 tx=0x7f9794009190 comp rx=0 tx=0).stop 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a415c1d0 msgr2=0x7f97a40c7d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a415c1d0 0x7f97a40c7d80 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f979c036040 tx=0x7f979c004a40 comp rx=0 tx=0).stop 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 shutdown_connections 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f978406b810 0x7f978406dcd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f97a415c1d0 0x7f97a40c7d80 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f97a40d0850 0x7f97a41f3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 --2- 172.21.15.161:0/2435504960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f97a40ce3c0 0x7f97a40b63b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 >> 172.21.15.161:0/2435504960 conn(0x7f97a409aa40 msgr2=0x7f97a40c66b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:44.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 shutdown_connections 2024-03-23T14:36:44.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.553+0000 7f97a89d7640 1 -- 172.21.15.161:0/2435504960 wait complete. 2024-03-23T14:36:44.560 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:36:44.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.667+0000 7f95ac883640 1 Processor -- start 2024-03-23T14:36:44.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.667+0000 7f95ac883640 1 -- start start 2024-03-23T14:36:44.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95ac883640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95a80b1720 0x7f95a80ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95ac883640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80cc540 0x7f95a80cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95ac883640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f95a80ca8e0 0x7f95a80cad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95ac883640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f95a80cf4b0 con 0x7f95a80b1720 2024-03-23T14:36:44.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95ac883640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f95a80cfd10 con 0x7f95a80ca8e0 2024-03-23T14:36:44.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95ac883640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f95a802c670 con 0x7f95a80cc540 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95a659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95a80b1720 0x7f95a80ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95a659c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f95a80b1720 0x7f95a80ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42006/0 (socket says 172.21.15.161:42006) 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95a659c640 1 -- 172.21.15.161:0/2325608447 learned_addr learned my addr 172.21.15.161:0/2325608447 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95a5d9b640 1 --2- 172.21.15.161:0/2325608447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80cc540 0x7f95a80cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.668+0000 7f95a6d9d640 1 --2- 172.21.15.161:0/2325608447 >> v2:172.21.15.183:3300/0 conn(0x7f95a80ca8e0 0x7f95a80cad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a659c640 1 -- 172.21.15.161:0/2325608447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80cc540 msgr2=0x7f95a80cd1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a659c640 1 --2- 172.21.15.161:0/2325608447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80cc540 0x7f95a80cd1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a659c640 1 -- 172.21.15.161:0/2325608447 >> v2:172.21.15.183:3300/0 conn(0x7f95a80ca8e0 msgr2=0x7f95a80cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a659c640 1 --2- 172.21.15.161:0/2325608447 >> v2:172.21.15.183:3300/0 conn(0x7f95a80ca8e0 0x7f95a80cad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a659c640 1 -- 172.21.15.161:0/2325608447 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95a80715a0 con 0x7f95a80b1720 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a5d9b640 1 --2- 172.21.15.161:0/2325608447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80cc540 0x7f95a80cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a659c640 1 --2- 172.21.15.161:0/2325608447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80b1720 0x7f95a80ccca0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f95a8071290 tx=0x7f959802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6122373c962998de server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.666 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a559a640 1 -- 172.21.15.161:0/2325608447 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959802ca70 con 0x7f95a80b1720 2024-03-23T14:36:44.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a559a640 1 -- 172.21.15.161:0/2325608447 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f959802cbd0 con 0x7f95a80b1720 2024-03-23T14:36:44.667 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.669+0000 7f95a559a640 1 -- 172.21.15.161:0/2325608447 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95980343f0 con 0x7f95a80b1720 2024-03-23T14:36:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 -- 172.21.15.161:0/2325608447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80b1720 msgr2=0x7f95a80ccca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.669 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 --2- 172.21.15.161:0/2325608447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80b1720 0x7f95a80ccca0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f95a8071290 tx=0x7f959802bed0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 -- 172.21.15.161:0/2325608447 shutdown_connections 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 --2- 172.21.15.161:0/2325608447 >> v2:172.21.15.183:3300/0 conn(0x7f95a80ca8e0 0x7f95a80cad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 --2- 172.21.15.161:0/2325608447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80cc540 0x7f95a80cd1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 --2- 172.21.15.161:0/2325608447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80b1720 0x7f95a80ccca0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 -- 172.21.15.161:0/2325608447 >> 172.21.15.161:0/2325608447 conn(0x7f95a80b1e60 msgr2=0x7f95a80b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 -- 172.21.15.161:0/2325608447 shutdown_connections 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.670+0000 7f95ac883640 1 -- 172.21.15.161:0/2325608447 wait complete. 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.671+0000 7f95ac883640 1 Processor -- start 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.671+0000 7f95ac883640 1 -- start start 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95ac883640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80b1720 0x7f95a81f4d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95ac883640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80ca8e0 0x7f95a8155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95ac883640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95a80cc540 0x7f95a8099920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95ac883640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f95a8155840 con 0x7f95a80ca8e0 2024-03-23T14:36:44.670 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95ac883640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f95a81f5310 con 0x7f95a80cc540 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95ac883640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f95a81f55f0 con 0x7f95a80b1720 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95a5d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80ca8e0 0x7f95a8155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95a5d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80ca8e0 0x7f95a8155300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42022/0 (socket says 172.21.15.161:42022) 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95a5d9b640 1 -- 172.21.15.161:0/319143682 learned_addr learned my addr 172.21.15.161:0/319143682 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95a659c640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80b1720 0x7f95a81f4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.672+0000 7f95a6d9d640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95a80cc540 0x7f95a8099920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95a5d9b640 1 -- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80b1720 msgr2=0x7f95a81f4d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95a5d9b640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80b1720 0x7f95a81f4d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95a5d9b640 1 -- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95a80cc540 msgr2=0x7f95a8099920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95a5d9b640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95a80cc540 0x7f95a8099920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95a5d9b640 1 -- 172.21.15.161:0/319143682 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959c009400 con 0x7f95a80ca8e0 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95a659c640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80b1720 0x7f95a81f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95a5d9b640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80ca8e0 0x7f95a8155300 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f959c00ebe0 tx=0x7f959c00c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f9596ffd640 1 -- 172.21.15.161:0/319143682 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c009760 con 0x7f95a80ca8e0 2024-03-23T14:36:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f9596ffd640 1 -- 172.21.15.161:0/319143682 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f959c00cc30 con 0x7f95a80ca8e0 2024-03-23T14:36:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9598009b30 con 0x7f95a80ca8e0 2024-03-23T14:36:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f9596ffd640 1 -- 172.21.15.161:0/319143682 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959c00eee0 con 0x7f95a80ca8e0 2024-03-23T14:36:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.673+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95a81f5bb0 con 0x7f95a80ca8e0 2024-03-23T14:36:44.672 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.674+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95a80cf680 con 0x7f95a80ca8e0 2024-03-23T14:36:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.674+0000 7f9596ffd640 1 -- 172.21.15.161:0/319143682 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f959c013470 con 0x7f95a80ca8e0 2024-03-23T14:36:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.675+0000 7f9596ffd640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f958806b840 0x7f958806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.675+0000 7f9596ffd640 1 -- 172.21.15.161:0/319143682 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f959c08a5e0 con 0x7f95a80ca8e0 2024-03-23T14:36:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.675+0000 7f95a659c640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f958806b840 0x7f958806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.675+0000 7f95a659c640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f958806b840 0x7f958806dd00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9598035000 tx=0x7f9598030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.679+0000 7f9596ffd640 1 -- 172.21.15.161:0/319143682 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f959c054f20 con 0x7f95a80ca8e0 2024-03-23T14:36:44.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.849+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f95a804f4e0 con 0x7f95a80ca8e0 2024-03-23T14:36:44.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.850+0000 7f9596ffd640 1 -- 172.21.15.161:0/319143682 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v36) ==== 100+0+11 (secure 0 0 0) 0x7f959c0585e0 con 0x7f95a80ca8e0 2024-03-23T14:36:44.846 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f958806b840 msgr2=0x7f958806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f958806b840 0x7f958806dd00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f9598035000 tx=0x7f9598030000 comp rx=0 tx=0).stop 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80ca8e0 msgr2=0x7f95a8155300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80ca8e0 0x7f95a8155300 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f959c00ebe0 tx=0x7f959c00c520 comp rx=0 tx=0).stop 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 shutdown_connections 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f958806b840 0x7f958806dd00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f95a80cc540 0x7f95a8099920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f95a80ca8e0 0x7f95a8155300 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 --2- 172.21.15.161:0/319143682 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f95a80b1720 0x7f95a81f4d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.848 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 >> 172.21.15.161:0/319143682 conn(0x7f95a80b1e60 msgr2=0x7f95a8161660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:44.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 shutdown_connections 2024-03-23T14:36:44.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.852+0000 7f95ac883640 1 -- 172.21.15.161:0/319143682 wait complete. 2024-03-23T14:36:44.858 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get data_new pg_num 2024-03-23T14:36:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.967+0000 7f1a76e47640 1 Processor -- start 2024-03-23T14:36:44.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.967+0000 7f1a76e47640 1 -- start start 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a76e47640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1a701549c0 0x7f1a7006e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a76e47640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a7015b780 0x7f1a7006d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a76e47640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a700b2c20 con 0x7f1a70070c20 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a76e47640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1a7006ee90 con 0x7f1a7015b780 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a76e47640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f1a700b2a10 con 0x7f1a701549c0 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a753e4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a7015b780 0x7f1a7006d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a753e4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1a7015b780 0x7f1a7006d6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42030/0 (socket says 172.21.15.161:42030) 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a753e4640 1 -- 172.21.15.161:0/2593453125 learned_addr learned my addr 172.21.15.161:0/2593453125 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a74be3640 1 --2- 172.21.15.161:0/2593453125 >> v2:172.21.15.183:3300/0 conn(0x7f1a701549c0 0x7f1a7006e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a6f7fe640 1 -- 172.21.15.161:0/2593453125 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3565138055 0 0) 0x7f1a700b2c20 con 0x7f1a70070c20 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a753e4640 1 -- 172.21.15.161:0/2593453125 >> v2:172.21.15.183:3300/0 conn(0x7f1a701549c0 msgr2=0x7f1a7006e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a753e4640 1 --2- 172.21.15.161:0/2593453125 >> v2:172.21.15.183:3300/0 conn(0x7f1a701549c0 0x7f1a7006e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a753e4640 1 -- 172.21.15.161:0/2593453125 >> v1:172.21.15.183:6789/0 conn(0x7f1a70070c20 legacy=0x7f1a7015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.968+0000 7f1a753e4640 1 -- 172.21.15.161:0/2593453125 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a700b2f00 con 0x7f1a7015b780 2024-03-23T14:36:44.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.969+0000 7f1a753e4640 1 --2- 172.21.15.161:0/2593453125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7015b780 0x7f1a7006d6f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1a700b2b50 tx=0x7f1a6802bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35f08fc3b11b9549 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.969+0000 7f1a6f7fe640 1 -- 172.21.15.161:0/2593453125 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a6802cb00 con 0x7f1a7015b780 2024-03-23T14:36:44.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.969+0000 7f1a6f7fe640 1 -- 172.21.15.161:0/2593453125 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a6802cc60 con 0x7f1a7015b780 2024-03-23T14:36:44.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.969+0000 7f1a6f7fe640 1 -- 172.21.15.161:0/2593453125 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a680343f0 con 0x7f1a7015b780 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.969+0000 7f1a76e47640 1 -- 172.21.15.161:0/2593453125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7015b780 msgr2=0x7f1a7006d6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.969+0000 7f1a76e47640 1 --2- 172.21.15.161:0/2593453125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7015b780 0x7f1a7006d6f0 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f1a700b2b50 tx=0x7f1a6802bf60 comp rx=0 tx=0).stop 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.970+0000 7f1a76e47640 1 -- 172.21.15.161:0/2593453125 shutdown_connections 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.970+0000 7f1a76e47640 1 --2- 172.21.15.161:0/2593453125 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a7015b780 0x7f1a7006d6f0 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.970+0000 7f1a76e47640 1 --2- 172.21.15.161:0/2593453125 >> v2:172.21.15.183:3300/0 conn(0x7f1a701549c0 0x7f1a7006e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.970+0000 7f1a76e47640 1 -- 172.21.15.161:0/2593453125 >> 172.21.15.161:0/2593453125 conn(0x7f1a7006b4a0 msgr2=0x7f1a7006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.970+0000 7f1a76e47640 1 -- 172.21.15.161:0/2593453125 shutdown_connections 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.970+0000 7f1a76e47640 1 -- 172.21.15.161:0/2593453125 wait complete. 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.971+0000 7f1a76e47640 1 Processor -- start 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.971+0000 7f1a76e47640 1 -- start start 2024-03-23T14:36:44.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.971+0000 7f1a76e47640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70070c20 0x7f1a7014a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a76e47640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a701549c0 0x7f1a700b09e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a76e47640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a7015b780 0x7f1a700b0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a76e47640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1a7014ad60 con 0x7f1a701549c0 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a76e47640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1a700ce690 con 0x7f1a70070c20 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a76e47640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1a700cd8d0 con 0x7f1a7015b780 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a701549c0 0x7f1a700b09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a74be3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70070c20 0x7f1a7014a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a701549c0 0x7f1a700b09e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42044/0 (socket says 172.21.15.161:42044) 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a74be3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70070c20 0x7f1a7014a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:36106/0 (socket says 172.21.15.161:36106) 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 -- 172.21.15.161:0/98949438 learned_addr learned my addr 172.21.15.161:0/98949438 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a753e4640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a7015b780 0x7f1a700b0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 -- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a7015b780 msgr2=0x7f1a700b0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a7015b780 0x7f1a700b0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 -- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70070c20 msgr2=0x7f1a7014a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70070c20 0x7f1a7014a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 -- 172.21.15.161:0/98949438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a68009b30 con 0x7f1a701549c0 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6ffff640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a701549c0 0x7f1a700b09e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c4b0 tx=0x7f1a5c00c970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6d7fa640 1 -- 172.21.15.161:0/98949438 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c005480 con 0x7f1a701549c0 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6d7fa640 1 -- 172.21.15.161:0/98949438 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a5c0035f0 con 0x7f1a701549c0 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a700cdbb0 con 0x7f1a701549c0 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a6d7fa640 1 -- 172.21.15.161:0/98949438 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a5c0039e0 con 0x7f1a701549c0 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.972+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a700ce170 con 0x7f1a701549c0 2024-03-23T14:36:44.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.973+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a70073570 con 0x7f1a701549c0 2024-03-23T14:36:44.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.974+0000 7f1a6d7fa640 1 -- 172.21.15.161:0/98949438 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1a5c003b40 con 0x7f1a701549c0 2024-03-23T14:36:44.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.974+0000 7f1a6d7fa640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1a5406b810 0x7f1a5406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:44.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.974+0000 7f1a6d7fa640 1 -- 172.21.15.161:0/98949438 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f1a5c089850 con 0x7f1a701549c0 2024-03-23T14:36:44.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.974+0000 7f1a74be3640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1a5406b810 0x7f1a5406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:44.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.975+0000 7f1a74be3640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1a5406b810 0x7f1a5406dcd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009b40 tx=0x7f1a64009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:44.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:44.978+0000 7f1a6d7fa640 1 -- 172.21.15.161:0/98949438 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1a5c054220 con 0x7f1a701549c0 2024-03-23T14:36:45.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.152+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"} v 0) -- 0x7f1a7004f550 con 0x7f1a701549c0 2024-03-23T14:36:45.148 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.152+0000 7f1a6d7fa640 1 -- 172.21.15.161:0/98949438 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "data_new", "var": "pg_num"}]=0 v36) ==== 97+0+11 (secure 0 0 0) 0x7f1a5c0578e0 con 0x7f1a701549c0 2024-03-23T14:36:45.148 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:36:45.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1a5406b810 msgr2=0x7f1a5406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1a5406b810 0x7f1a5406dcd0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f1a64009b40 tx=0x7f1a64009190 comp rx=0 tx=0).stop 2024-03-23T14:36:45.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a701549c0 msgr2=0x7f1a700b09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a701549c0 0x7f1a700b09e0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f1a5c00c4b0 tx=0x7f1a5c00c970 comp rx=0 tx=0).stop 2024-03-23T14:36:45.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 shutdown_connections 2024-03-23T14:36:45.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1a5406b810 0x7f1a5406dcd0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1a7015b780 0x7f1a700b0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.150 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1a701549c0 0x7f1a700b09e0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 --2- 172.21.15.161:0/98949438 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1a70070c20 0x7f1a7014a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 >> 172.21.15.161:0/98949438 conn(0x7f1a7006b4a0 msgr2=0x7f1a7006b880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:45.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 shutdown_connections 2024-03-23T14:36:45.151 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.154+0000 7f1a76e47640 1 -- 172.21.15.161:0/98949438 wait complete. 2024-03-23T14:36:45.161 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:36:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.264+0000 7fec90763640 1 Processor -- start 2024-03-23T14:36:45.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.264+0000 7fec90763640 1 -- start start 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec90763640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b4260 0x7fec8815b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec90763640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fec880b3610 0x7fec880b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec90763640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec88070b90 con 0x7fec8806eb30 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec90763640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fec880b3fe0 con 0x7fec880b3610 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec90763640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fec8815ab90 con 0x7fec880b4260 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec8dcfe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b4260 0x7fec8815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec8ed00640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fec880b3610 0x7fec880b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec8e4ff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fec8806eb30 0x7fec880b4870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:34018/0 (socket says 172.21.15.161:34018) 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec8ed00640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fec880b3610 0x7fec880b3aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:36114/0 (socket says 172.21.15.161:36114) 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec8dcfe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b4260 0x7fec8815b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:54284/0 (socket says 172.21.15.161:54284) 2024-03-23T14:36:45.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.265+0000 7fec8e4ff640 1 -- 172.21.15.161:0/3822148879 learned_addr learned my addr 172.21.15.161:0/3822148879 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.266+0000 7fec8ed00640 1 -- 172.21.15.161:0/3822148879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b4260 msgr2=0x7fec8815b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.266+0000 7fec8ed00640 1 --2- 172.21.15.161:0/3822148879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b4260 0x7fec8815b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.266+0000 7fec8ed00640 1 -- 172.21.15.161:0/3822148879 >> v1:172.21.15.183:6789/0 conn(0x7fec8806eb30 legacy=0x7fec880b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.266+0000 7fec8ed00640 1 -- 172.21.15.161:0/3822148879 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec8815b0d0 con 0x7fec880b3610 2024-03-23T14:36:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.266+0000 7fec8d4fd640 1 -- 172.21.15.161:0/3822148879 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 693429664 0 0) 0x7fec88070b90 con 0x7fec8806eb30 2024-03-23T14:36:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.266+0000 7fec8dcfe640 1 --2- 172.21.15.161:0/3822148879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b4260 0x7fec8815b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.266+0000 7fec8ed00640 1 --2- 172.21.15.161:0/3822148879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec880b3610 0x7fec880b3aa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fec800097a0 tx=0x7fec8002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c05b13ecb5891da9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec8d4fd640 1 -- 172.21.15.161:0/3822148879 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80031830 con 0x7fec880b3610 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec8d4fd640 1 -- 172.21.15.161:0/3822148879 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec80031990 con 0x7fec880b3610 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec8d4fd640 1 -- 172.21.15.161:0/3822148879 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec80031d20 con 0x7fec880b3610 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec90763640 1 -- 172.21.15.161:0/3822148879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec880b3610 msgr2=0x7fec880b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec90763640 1 --2- 172.21.15.161:0/3822148879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec880b3610 0x7fec880b3aa0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fec800097a0 tx=0x7fec8002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec90763640 1 -- 172.21.15.161:0/3822148879 shutdown_connections 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec90763640 1 --2- 172.21.15.161:0/3822148879 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec880b3610 0x7fec880b3aa0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec90763640 1 --2- 172.21.15.161:0/3822148879 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b4260 0x7fec8815b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec90763640 1 -- 172.21.15.161:0/3822148879 >> 172.21.15.161:0/3822148879 conn(0x7fec880b17f0 msgr2=0x7fec880b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec90763640 1 -- 172.21.15.161:0/3822148879 shutdown_connections 2024-03-23T14:36:45.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.267+0000 7fec90763640 1 -- 172.21.15.161:0/3822148879 wait complete. 2024-03-23T14:36:45.266 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.268+0000 7fec90763640 1 Processor -- start 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.268+0000 7fec90763640 1 -- start start 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec90763640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec8806eb30 0x7fec88143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec90763640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b3610 0x7fec88145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec90763640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec880b4260 0x7fec880ce840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec90763640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fec88144b20 con 0x7fec880b4260 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec90763640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fec88148800 con 0x7fec8806eb30 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec90763640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fec881444f0 con 0x7fec880b3610 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec8ed00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec880b4260 0x7fec880ce840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec8ed00640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec880b4260 0x7fec880ce840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42056/0 (socket says 172.21.15.161:42056) 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec8e4ff640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec8806eb30 0x7fec88143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec8ed00640 1 -- 172.21.15.161:0/1991499251 learned_addr learned my addr 172.21.15.161:0/1991499251 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec8dcfe640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b3610 0x7fec88145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec8e4ff640 1 -- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b3610 msgr2=0x7fec88145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.269+0000 7fec8e4ff640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b3610 0x7fec88145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec8e4ff640 1 -- 172.21.15.161:0/1991499251 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec880b4260 msgr2=0x7fec880ce840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec8e4ff640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec880b4260 0x7fec880ce840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec8e4ff640 1 -- 172.21.15.161:0/1991499251 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec78009400 con 0x7fec8806eb30 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec8ed00640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec880b4260 0x7fec880ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec8e4ff640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec8806eb30 0x7fec88143c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec8400c7c0 tx=0x7fec8400cc80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec7effd640 1 -- 172.21.15.161:0/1991499251 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8400a640 con 0x7fec8806eb30 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec7effd640 1 -- 172.21.15.161:0/1991499251 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec8400a7c0 con 0x7fec8806eb30 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec80009400 con 0x7fec8806eb30 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec7effd640 1 -- 172.21.15.161:0/1991499251 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec8400ab70 con 0x7fec8806eb30 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.270+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec880cf5c0 con 0x7fec8806eb30 2024-03-23T14:36:45.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.271+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fec88144c60 con 0x7fec8806eb30 2024-03-23T14:36:45.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.271+0000 7fec7effd640 1 -- 172.21.15.161:0/1991499251 <== mon.1 v2:172.21.15.183:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fec8400acd0 con 0x7fec8806eb30 2024-03-23T14:36:45.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.272+0000 7fec7effd640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fec6c06b810 0x7fec6c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.272+0000 7fec7effd640 1 -- 172.21.15.161:0/1991499251 <== mon.1 v2:172.21.15.183:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fec84089ab0 con 0x7fec8806eb30 2024-03-23T14:36:45.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.272+0000 7fec8dcfe640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fec6c06b810 0x7fec6c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.272+0000 7fec8dcfe640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fec6c06b810 0x7fec6c06dcd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fec78009c30 tx=0x7fec78009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.280+0000 7fec7effd640 1 -- 172.21.15.161:0/1991499251 <== mon.1 v2:172.21.15.183:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fec84054420 con 0x7fec8806eb30 2024-03-23T14:36:45.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.487+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fec880b5de0 con 0x7fec8806eb30 2024-03-23T14:36:45.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.488+0000 7fec7effd640 1 -- 172.21.15.161:0/1991499251 <== mon.1 v2:172.21.15.183:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5182 (secure 0 0 0) 0x7fec84018020 con 0x7fec8806eb30 2024-03-23T14:36:45.484 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:45.484 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/626265302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":626265302},{"type":"v1","addr":"172.21.15.183:6837","nonce":626265302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5220,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/208344896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":208344896},{"type":"v1","addr":"172.21.15.161:6833","nonce":208344896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/82303731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":82303731},{"type":"v1","addr":"172.21.15.161:6835","nonce":82303731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:06.267736+0000","modified":"2024-03-23T14:36:19.582504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5235":{"gid":5235,"name":"d","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6836/2372368263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2372368263},{"type":"v1","addr":"172.21.15.183:6836","nonce":2372368263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5235,"qdb_cluster":[5235]},"id":2}]} 2024-03-23T14:36:45.485 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 16 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fec6c06b810 msgr2=0x7fec6c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fec6c06b810 0x7fec6c06dcd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fec78009c30 tx=0x7fec78009190 comp rx=0 tx=0).stop 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec8806eb30 msgr2=0x7fec88143c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec8806eb30 0x7fec88143c80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fec8400c7c0 tx=0x7fec8400cc80 comp rx=0 tx=0).stop 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 shutdown_connections 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fec6c06b810 0x7fec6c06dcd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fec880b4260 0x7fec880ce840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fec880b3610 0x7fec88145fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.486 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 --2- 172.21.15.161:0/1991499251 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fec8806eb30 0x7fec88143c80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 >> 172.21.15.161:0/1991499251 conn(0x7fec880b17f0 msgr2=0x7fec88165a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:45.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 shutdown_connections 2024-03-23T14:36:45.487 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.490+0000 7fec90763640 1 -- 172.21.15.161:0/1991499251 wait complete. 2024-03-23T14:36:45.497 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:45.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.601+0000 7f0e73cc3640 1 Processor -- start 2024-03-23T14:36:45.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.601+0000 7f0e73cc3640 1 -- start start 2024-03-23T14:36:45.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e73cc3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f0e6c0b3610 0x7f0e6c0b3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e73cc3640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e6c0b3fa0 con 0x7f0e6c06eb30 2024-03-23T14:36:45.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e73cc3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e6c070b90 con 0x7f0e6c0b4260 2024-03-23T14:36:45.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e73cc3640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f0e6c15ab90 con 0x7f0e6c0b3610 2024-03-23T14:36:45.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e71a5f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f0e6c06eb30 0x7f0e6c0b4870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35204/0 (socket says 172.21.15.161:35204) 2024-03-23T14:36:45.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e71a5f640 1 -- 172.21.15.161:0/1726295345 learned_addr learned my addr 172.21.15.161:0/1726295345 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:45.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e72260640 1 --2- 172.21.15.161:0/1726295345 >> v2:172.21.15.183:3300/0 conn(0x7f0e6c0b3610 0x7f0e6c0b3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e70a5d640 1 -- 172.21.15.161:0/1726295345 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1930561072 0 0) 0x7f0e68001490 con 0x7f0e6c06eb30 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.602+0000 7f0e70a5d640 1 -- 172.21.15.161:0/1726295345 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e58003520 con 0x7f0e6c06eb30 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.603+0000 7f0e72260640 1 -- 172.21.15.161:0/1726295345 >> v1:172.21.15.183:6789/0 conn(0x7f0e6c0b4260 legacy=0x7f0e6c15b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.603+0000 7f0e70a5d640 1 -- 172.21.15.161:0/1726295345 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553063250 0 0) 0x7f0e6c070b90 con 0x7f0e6c0b4260 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.603+0000 7f0e72260640 1 -- 172.21.15.161:0/1726295345 >> v1:172.21.15.161:6789/0 conn(0x7f0e6c06eb30 legacy=0x7f0e6c0b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.603+0000 7f0e72260640 1 -- 172.21.15.161:0/1726295345 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e6c15b0d0 con 0x7f0e6c0b3610 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.603+0000 7f0e72260640 1 --2- 172.21.15.161:0/1726295345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 0x7f0e6c0b3a60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0e640097a0 tx=0x7f0e6402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd58edb33d5aa18 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.603+0000 7f0e70a5d640 1 -- 172.21.15.161:0/1726295345 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64031830 con 0x7f0e6c0b3610 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e70a5d640 1 -- 172.21.15.161:0/1726295345 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e64031990 con 0x7f0e6c0b3610 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e70a5d640 1 -- 172.21.15.161:0/1726295345 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e64031d20 con 0x7f0e6c0b3610 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e73cc3640 1 -- 172.21.15.161:0/1726295345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 msgr2=0x7f0e6c0b3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e73cc3640 1 --2- 172.21.15.161:0/1726295345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 0x7f0e6c0b3a60 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0e640097a0 tx=0x7f0e6402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e73cc3640 1 -- 172.21.15.161:0/1726295345 shutdown_connections 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e73cc3640 1 --2- 172.21.15.161:0/1726295345 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 0x7f0e6c0b3a60 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e73cc3640 1 -- 172.21.15.161:0/1726295345 >> 172.21.15.161:0/1726295345 conn(0x7f0e6c0b17f0 msgr2=0x7f0e6c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:45.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e73cc3640 1 -- 172.21.15.161:0/1726295345 shutdown_connections 2024-03-23T14:36:45.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.604+0000 7f0e73cc3640 1 -- 172.21.15.161:0/1726295345 wait complete. 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.605+0000 7f0e73cc3640 1 Processor -- start 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.605+0000 7f0e73cc3640 1 -- start start 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e73cc3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e6c06eb30 0x7f0e6c147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e73cc3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 0x7f0e6c06d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e73cc3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e6c0b4260 0x7f0e6c146540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e73cc3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c0ce220 con 0x7f0e6c06eb30 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e73cc3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f0e6c147a90 con 0x7f0e6c0b3610 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e73cc3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f0e6c147d70 con 0x7f0e6c0b4260 2024-03-23T14:36:45.607 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e71a5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e6c06eb30 0x7f0e6c147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e71a5f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e6c06eb30 0x7f0e6c147190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42070/0 (socket says 172.21.15.161:42070) 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e71a5f640 1 -- 172.21.15.161:0/989749762 learned_addr learned my addr 172.21.15.161:0/989749762 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e7125e640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 0x7f0e6c06d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e72260640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e6c0b4260 0x7f0e6c146540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e71a5f640 1 -- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e6c0b4260 msgr2=0x7f0e6c146540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e71a5f640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e6c0b4260 0x7f0e6c146540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e71a5f640 1 -- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 msgr2=0x7f0e6c06d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e71a5f640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 0x7f0e6c06d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.606+0000 7f0e71a5f640 1 -- 172.21.15.161:0/989749762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e64009400 con 0x7f0e6c06eb30 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.607+0000 7f0e71a5f640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e6c06eb30 0x7f0e6c147190 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800cb40 tx=0x7f0e6800a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.607+0000 7f0e627fc640 1 -- 172.21.15.161:0/989749762 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e6800aca0 con 0x7f0e6c06eb30 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.607+0000 7f0e627fc640 1 -- 172.21.15.161:0/989749762 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e6800ae20 con 0x7f0e6c06eb30 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.607+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e6c148050 con 0x7f0e6c06eb30 2024-03-23T14:36:45.608 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.607+0000 7f0e627fc640 1 -- 172.21.15.161:0/989749762 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e68012640 con 0x7f0e6c06eb30 2024-03-23T14:36:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.607+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e6c1476d0 con 0x7f0e6c06eb30 2024-03-23T14:36:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.608+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e6c0ce6d0 con 0x7f0e6c06eb30 2024-03-23T14:36:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.608+0000 7f0e627fc640 1 -- 172.21.15.161:0/989749762 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f0e68007070 con 0x7f0e6c06eb30 2024-03-23T14:36:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.608+0000 7f0e627fc640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e5006b840 0x7f0e5006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.609+0000 7f0e627fc640 1 -- 172.21.15.161:0/989749762 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f0e6808a0b0 con 0x7f0e6c06eb30 2024-03-23T14:36:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.609+0000 7f0e7125e640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e5006b840 0x7f0e5006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.609+0000 7f0e7125e640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e5006b840 0x7f0e5006dd00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c16b760 tx=0x7f0e5c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.609 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.613+0000 7f0e627fc640 1 -- 172.21.15.161:0/989749762 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f0e6c0ce6d0 con 0x7f0e6c06eb30 2024-03-23T14:36:45.781 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.785+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0e6c0b3fa0 con 0x7f0e6c06eb30 2024-03-23T14:36:45.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.787+0000 7f0e627fc640 1 -- 172.21.15.161:0/989749762 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7f0e6c0b3fa0 con 0x7f0e6c06eb30 2024-03-23T14:36:45.783 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:45.783 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.788+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e5006b840 msgr2=0x7f0e5006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.788+0000 7f0e73cc3640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e5006b840 0x7f0e5006dd00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c16b760 tx=0x7f0e5c009040 comp rx=0 tx=0).stop 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.788+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e6c06eb30 msgr2=0x7f0e6c147190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.788+0000 7f0e73cc3640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e6c06eb30 0x7f0e6c147190 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f0e6800cb40 tx=0x7f0e6800a060 comp rx=0 tx=0).stop 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.789+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 shutdown_connections 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.789+0000 7f0e73cc3640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f0e5006b840 0x7f0e5006dd00 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.789+0000 7f0e73cc3640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f0e6c0b4260 0x7f0e6c146540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.789+0000 7f0e73cc3640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f0e6c0b3610 0x7f0e6c06d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.789+0000 7f0e73cc3640 1 --2- 172.21.15.161:0/989749762 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f0e6c06eb30 0x7f0e6c147190 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.789+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 >> 172.21.15.161:0/989749762 conn(0x7f0e6c0b17f0 msgr2=0x7f0e6c15e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:45.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.789+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 shutdown_connections 2024-03-23T14:36:45.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.789+0000 7f0e73cc3640 1 -- 172.21.15.161:0/989749762 wait complete. 2024-03-23T14:36:45.796 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T14:36:45.796 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:36:45.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.901+0000 7f42eb888640 1 Processor -- start 2024-03-23T14:36:45.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.901+0000 7f42eb888640 1 -- start start 2024-03-23T14:36:45.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42eb888640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f42e40724e0 0x7f42e406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42eb888640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e415be70 con 0x7f42e4072100 2024-03-23T14:36:45.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42eb888640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42e415c150 con 0x7f42e406ead0 2024-03-23T14:36:45.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42eb888640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f42e406ef80 con 0x7f42e40724e0 2024-03-23T14:36:45.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42e9624640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f42e4072100 0x7f42e415b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35216/0 (socket says 172.21.15.161:35216) 2024-03-23T14:36:45.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42e9624640 1 -- 172.21.15.161:0/3901716677 learned_addr learned my addr 172.21.15.161:0/3901716677 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:45.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42e8e23640 1 --2- 172.21.15.161:0/3901716677 >> v2:172.21.15.183:3300/0 conn(0x7f42e40724e0 0x7f42e406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42dbfff640 1 -- 172.21.15.161:0/3901716677 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3598858118 0 0) 0x7f42e415be70 con 0x7f42e4072100 2024-03-23T14:36:45.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42dbfff640 1 -- 172.21.15.161:0/3901716677 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42d0003520 con 0x7f42e4072100 2024-03-23T14:36:45.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42dbfff640 1 -- 172.21.15.161:0/3901716677 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1737432249 0 0) 0x7f42e415c150 con 0x7f42e406ead0 2024-03-23T14:36:45.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42dbfff640 1 -- 172.21.15.161:0/3901716677 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f42e415be70 con 0x7f42e406ead0 2024-03-23T14:36:45.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42e8e23640 1 -- 172.21.15.161:0/3901716677 >> v1:172.21.15.183:6789/0 conn(0x7f42e406ead0 legacy=0x7f42e40b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42e8e23640 1 -- 172.21.15.161:0/3901716677 >> v1:172.21.15.161:6789/0 conn(0x7f42e4072100 legacy=0x7f42e415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.902+0000 7f42e8e23640 1 -- 172.21.15.161:0/3901716677 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42e40b2c70 con 0x7f42e40724e0 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.903+0000 7f42e8e23640 1 --2- 172.21.15.161:0/3901716677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e40724e0 0x7f42e406e590 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f42d40097a0 tx=0x7f42d402b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cd99fadf39ef403 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.903+0000 7f42dbfff640 1 -- 172.21.15.161:0/3901716677 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4030610 con 0x7f42e40724e0 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.903+0000 7f42dbfff640 1 -- 172.21.15.161:0/3901716677 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d4030770 con 0x7f42e40724e0 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.903+0000 7f42dbfff640 1 -- 172.21.15.161:0/3901716677 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4030b00 con 0x7f42e40724e0 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.904+0000 7f42eb888640 1 -- 172.21.15.161:0/3901716677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e40724e0 msgr2=0x7f42e406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.904+0000 7f42eb888640 1 --2- 172.21.15.161:0/3901716677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e40724e0 0x7f42e406e590 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f42d40097a0 tx=0x7f42d402b960 comp rx=0 tx=0).stop 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.904+0000 7f42eb888640 1 -- 172.21.15.161:0/3901716677 shutdown_connections 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.904+0000 7f42eb888640 1 --2- 172.21.15.161:0/3901716677 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e40724e0 0x7f42e406e590 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.904+0000 7f42eb888640 1 -- 172.21.15.161:0/3901716677 >> 172.21.15.161:0/3901716677 conn(0x7f42e40b17f0 msgr2=0x7f42e40b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.904+0000 7f42eb888640 1 -- 172.21.15.161:0/3901716677 shutdown_connections 2024-03-23T14:36:45.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.904+0000 7f42eb888640 1 -- 172.21.15.161:0/3901716677 wait complete. 2024-03-23T14:36:45.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.905+0000 7f42eb888640 1 Processor -- start 2024-03-23T14:36:45.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.905+0000 7f42eb888640 1 -- start start 2024-03-23T14:36:45.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42eb888640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e406ead0 0x7f42e41e6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42eb888640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42e4072100 0x7f42e414ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42eb888640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42e40724e0 0x7f42e414f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42eb888640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41e5c50 con 0x7f42e4072100 2024-03-23T14:36:45.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42eb888640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f42e41e5040 con 0x7f42e406ead0 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42eb888640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f42e41e5320 con 0x7f42e40724e0 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42e4072100 0x7f42e414ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42e4072100 0x7f42e414ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42082/0 (socket says 172.21.15.161:42082) 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 -- 172.21.15.161:0/1179144918 learned_addr learned my addr 172.21.15.161:0/1179144918 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e9624640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e406ead0 0x7f42e41e6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e9e25640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42e40724e0 0x7f42e414f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 -- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42e40724e0 msgr2=0x7f42e414f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42e40724e0 0x7f42e414f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 -- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e406ead0 msgr2=0x7f42e41e6ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e406ead0 0x7f42e41e6ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 -- 172.21.15.161:0/1179144918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d4009400 con 0x7f42e4072100 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42e8e23640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42e4072100 0x7f42e414ec10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f42d402bf70 tx=0x7f42d40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42d9ffb640 1 -- 172.21.15.161:0/1179144918 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d403e070 con 0x7f42e4072100 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42d9ffb640 1 -- 172.21.15.161:0/1179144918 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42d40397a0 con 0x7f42e4072100 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42e41e5600 con 0x7f42e4072100 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.906+0000 7f42d9ffb640 1 -- 172.21.15.161:0/1179144918 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42d4039b70 con 0x7f42e4072100 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.907+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42e41e70e0 con 0x7f42e4072100 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.907+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42e40735e0 con 0x7f42e4072100 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.908+0000 7f42d9ffb640 1 -- 172.21.15.161:0/1179144918 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42d4039cd0 con 0x7f42e4072100 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.908+0000 7f42d9ffb640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42c806b810 0x7f42c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.908+0000 7f42d9ffb640 1 -- 172.21.15.161:0/1179144918 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f42d40ac550 con 0x7f42e4072100 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.909+0000 7f42e9624640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42c806b810 0x7f42c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:45.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.909+0000 7f42e9624640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42c806b810 0x7f42c806dcd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f42dc000f60 tx=0x7f42dc007d70 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:45.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:45.913+0000 7f42d9ffb640 1 -- 172.21.15.161:0/1179144918 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f42d4076e10 con 0x7f42e4072100 2024-03-23T14:36:46.118 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.123+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42e404f550 con 0x7f42e4072100 2024-03-23T14:36:46.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.123+0000 7f42d9ffb640 1 -- 172.21.15.161:0/1179144918 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5182 (secure 0 0 0) 0x7f42d407a4d0 con 0x7f42e4072100 2024-03-23T14:36:46.119 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:46.119 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/626265302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":626265302},{"type":"v1","addr":"172.21.15.183:6837","nonce":626265302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5220,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/208344896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":208344896},{"type":"v1","addr":"172.21.15.161:6833","nonce":208344896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/82303731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":82303731},{"type":"v1","addr":"172.21.15.161:6835","nonce":82303731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:06.267736+0000","modified":"2024-03-23T14:36:19.582504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5235":{"gid":5235,"name":"d","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6836/2372368263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2372368263},{"type":"v1","addr":"172.21.15.183:6836","nonce":2372368263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5235,"qdb_cluster":[5235]},"id":2}]} 2024-03-23T14:36:46.120 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 16 2024-03-23T14:36:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42c806b810 msgr2=0x7f42c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42c806b810 0x7f42c806dcd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f42dc000f60 tx=0x7f42dc007d70 comp rx=0 tx=0).stop 2024-03-23T14:36:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42e4072100 msgr2=0x7f42e414ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42e4072100 0x7f42e414ec10 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f42d402bf70 tx=0x7f42d40046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 shutdown_connections 2024-03-23T14:36:46.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42c806b810 0x7f42c806dcd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42e40724e0 0x7f42e414f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42e4072100 0x7f42e414ec10 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 --2- 172.21.15.161:0/1179144918 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42e406ead0 0x7f42e41e6ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 >> 172.21.15.161:0/1179144918 conn(0x7f42e40b17f0 msgr2=0x7f42e40cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:46.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 shutdown_connections 2024-03-23T14:36:46.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.125+0000 7f42eb888640 1 -- 172.21.15.161:0/1179144918 wait complete. 2024-03-23T14:36:46.132 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:36:46.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.236+0000 7fa42cfb5640 1 Processor -- start 2024-03-23T14:36:46.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.236+0000 7fa42cfb5640 1 -- start start 2024-03-23T14:36:46.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.237+0000 7fa42cfb5640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa428145080 0x7fa428072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.237+0000 7fa42cfb5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa4281601f0 0x7fa42815c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.237+0000 7fa42cfb5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4280cf4f0 con 0x7fa4281607d0 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.237+0000 7fa42cfb5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa428071010 con 0x7fa4281601f0 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.237+0000 7fa42cfb5640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa4280cef30 con 0x7fa428145080 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.237+0000 7fa426d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa4281601f0 0x7fa42815c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa426d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa4281601f0 0x7fa42815c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42084/0 (socket says 172.21.15.161:42084) 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa426d9d640 1 -- 172.21.15.161:0/4259437743 learned_addr learned my addr 172.21.15.161:0/4259437743 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa425d9b640 1 --1- 172.21.15.161:0/4259437743 >> v1:172.21.15.161:6789/0 conn(0x7fa4281607d0 0x7fa42815fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35218/0 (socket says 172.21.15.161:35218) 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa42659c640 1 --2- 172.21.15.161:0/4259437743 >> v2:172.21.15.183:3300/0 conn(0x7fa428145080 0x7fa428072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa426d9d640 1 -- 172.21.15.161:0/4259437743 >> v2:172.21.15.183:3300/0 conn(0x7fa428145080 msgr2=0x7fa428072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa426d9d640 1 --2- 172.21.15.161:0/4259437743 >> v2:172.21.15.183:3300/0 conn(0x7fa428145080 0x7fa428072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa426d9d640 1 -- 172.21.15.161:0/4259437743 >> v1:172.21.15.161:6789/0 conn(0x7fa4281607d0 legacy=0x7fa42815fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa426d9d640 1 -- 172.21.15.161:0/4259437743 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4280cf210 con 0x7fa4281601f0 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.238+0000 7fa42559a640 1 -- 172.21.15.161:0/4259437743 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1481575297 0 0) 0x7fa4280cf4f0 con 0x7fa4281607d0 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa426d9d640 1 --2- 172.21.15.161:0/4259437743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281601f0 0x7fa42815c4f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa42806c9a0 tx=0x7fa42002bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d53684128770dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42559a640 1 -- 172.21.15.161:0/4259437743 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42002ca80 con 0x7fa4281601f0 2024-03-23T14:36:46.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42559a640 1 -- 172.21.15.161:0/4259437743 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa42002cbe0 con 0x7fa4281601f0 2024-03-23T14:36:46.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42559a640 1 -- 172.21.15.161:0/4259437743 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4200343f0 con 0x7fa4281601f0 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4259437743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281601f0 msgr2=0x7fa42815c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4259437743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281601f0 0x7fa42815c4f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fa42806c9a0 tx=0x7fa42002bee0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4259437743 shutdown_connections 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4259437743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281601f0 0x7fa42815c4f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4259437743 >> v2:172.21.15.183:3300/0 conn(0x7fa428145080 0x7fa428072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4259437743 >> 172.21.15.161:0/4259437743 conn(0x7fa4280b17f0 msgr2=0x7fa4280b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4259437743 shutdown_connections 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.239+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4259437743 wait complete. 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.241+0000 7fa42cfb5640 1 Processor -- start 2024-03-23T14:36:46.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.241+0000 7fa42cfb5640 1 -- start start 2024-03-23T14:36:46.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.241+0000 7fa42cfb5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428145080 0x7fa4281f48e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.241+0000 7fa42cfb5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4281601f0 0x7fa428071330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.241+0000 7fa42cfb5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281607d0 0x7fa428071870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.241+0000 7fa42cfb5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa4281571e0 con 0x7fa4281607d0 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.241+0000 7fa42cfb5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa4281574c0 con 0x7fa4281601f0 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.241+0000 7fa42cfb5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa4281577a0 con 0x7fa428145080 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281607d0 0x7fa428071870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281607d0 0x7fa428071870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42088/0 (socket says 172.21.15.161:42088) 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 -- 172.21.15.161:0/4037189545 learned_addr learned my addr 172.21.15.161:0/4037189545 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa42659c640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428145080 0x7fa4281f48e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa425d9b640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4281601f0 0x7fa428071330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 -- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428145080 msgr2=0x7fa4281f48e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428145080 0x7fa4281f48e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 -- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4281601f0 msgr2=0x7fa428071330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4281601f0 0x7fa428071330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 -- 172.21.15.161:0/4037189545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa420009b30 con 0x7fa4281607d0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa426d9d640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281607d0 0x7fa428071870 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa420036040 tx=0x7fa420004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa416ffd640 1 -- 172.21.15.161:0/4037189545 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420009510 con 0x7fa4281607d0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa416ffd640 1 -- 172.21.15.161:0/4037189545 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa420003990 con 0x7fa4281607d0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa428157a80 con 0x7fa4281607d0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa416ffd640 1 -- 172.21.15.161:0/4037189545 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa420003b20 con 0x7fa4281607d0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.242+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa428140bd0 con 0x7fa4281607d0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.243+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa428157f40 con 0x7fa4281607d0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.243+0000 7fa416ffd640 1 -- 172.21.15.161:0/4037189545 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa42002cd50 con 0x7fa4281607d0 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.244+0000 7fa416ffd640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40806b810 0x7fa40806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.244+0000 7fa416ffd640 1 -- 172.21.15.161:0/4037189545 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fa4200ac440 con 0x7fa4281607d0 2024-03-23T14:36:46.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.244+0000 7fa42659c640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40806b810 0x7fa40806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.244+0000 7fa42659c640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40806b810 0x7fa40806dcd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009b40 tx=0x7fa41c009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.248+0000 7fa416ffd640 1 -- 172.21.15.161:0/4037189545 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa420076d80 con 0x7fa4281607d0 2024-03-23T14:36:46.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.453+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa42806cb70 con 0x7fa4281607d0 2024-03-23T14:36:46.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.454+0000 7fa416ffd640 1 -- 172.21.15.161:0/4037189545 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+5182 (secure 0 0 0) 0x7fa42007a440 con 0x7fa4281607d0 2024-03-23T14:36:46.450 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:46.450 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":16,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/626265302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":626265302},{"type":"v1","addr":"172.21.15.183:6837","nonce":626265302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5220,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/208344896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":208344896},{"type":"v1","addr":"172.21.15.161:6833","nonce":208344896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/82303731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":82303731},{"type":"v1","addr":"172.21.15.161:6835","nonce":82303731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:06.267736+0000","modified":"2024-03-23T14:36:19.582504+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5235},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5235":{"gid":5235,"name":"d","rank":0,"incarnation":11,"state":"up:active","state_seq":3,"addr":"172.21.15.183:6836/2372368263","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2372368263},{"type":"v1","addr":"172.21.15.183:6836","nonce":2372368263}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[5,6],"metadata_pool":4,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5235,"qdb_cluster":[5235]},"id":2}]} 2024-03-23T14:36:46.450 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 16 2024-03-23T14:36:46.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40806b810 msgr2=0x7fa40806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40806b810 0x7fa40806dcd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fa41c009b40 tx=0x7fa41c009190 comp rx=0 tx=0).stop 2024-03-23T14:36:46.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281607d0 msgr2=0x7fa428071870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281607d0 0x7fa428071870 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fa420036040 tx=0x7fa420004a40 comp rx=0 tx=0).stop 2024-03-23T14:36:46.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 shutdown_connections 2024-03-23T14:36:46.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40806b810 0x7fa40806dcd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4281607d0 0x7fa428071870 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4281601f0 0x7fa428071330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 --2- 172.21.15.161:0/4037189545 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428145080 0x7fa4281f48e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 >> 172.21.15.161:0/4037189545 conn(0x7fa4280b17f0 msgr2=0x7fa4280b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:46.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 shutdown_connections 2024-03-23T14:36:46.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.456+0000 7fa42cfb5640 1 -- 172.21.15.161:0/4037189545 wait complete. 2024-03-23T14:36:46.463 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:46.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.568+0000 7fd48bbf4640 1 Processor -- start 2024-03-23T14:36:46.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.568+0000 7fd48bbf4640 1 -- start start 2024-03-23T14:36:46.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48bbf4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd484072100 0x7fd48415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48bbf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd4840b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48bbf4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd48415be70 con 0x7fd4840724e0 2024-03-23T14:36:46.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48bbf4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fd48415c430 con 0x7fd484072100 2024-03-23T14:36:46.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48bbf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd48415c150 con 0x7fd48406eca0 2024-03-23T14:36:46.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48918f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd4840724e0 0x7fd48406e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35226/0 (socket says 172.21.15.161:35226) 2024-03-23T14:36:46.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48918f640 1 -- 172.21.15.161:0/3684663180 learned_addr learned my addr 172.21.15.161:0/3684663180 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:46.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48a191640 1 --2- 172.21.15.161:0/3684663180 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd4840b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd489990640 1 --2- 172.21.15.161:0/3684663180 >> v2:172.21.15.183:3300/0 conn(0x7fd484072100 0x7fd48415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48898e640 1 -- 172.21.15.161:0/3684663180 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3498679960 0 0) 0x7fd48415be70 con 0x7fd4840724e0 2024-03-23T14:36:46.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48a191640 1 -- 172.21.15.161:0/3684663180 >> v2:172.21.15.183:3300/0 conn(0x7fd484072100 msgr2=0x7fd48415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:46.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48a191640 1 --2- 172.21.15.161:0/3684663180 >> v2:172.21.15.183:3300/0 conn(0x7fd484072100 0x7fd48415b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48a191640 1 -- 172.21.15.161:0/3684663180 >> v1:172.21.15.161:6789/0 conn(0x7fd4840724e0 legacy=0x7fd48406e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd48a191640 1 -- 172.21.15.161:0/3684663180 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd4840b2c70 con 0x7fd48406eca0 2024-03-23T14:36:46.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.569+0000 7fd489990640 1 --2- 172.21.15.161:0/3684663180 >> v2:172.21.15.183:3300/0 conn(0x7fd484072100 0x7fd48415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:46.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.570+0000 7fd48a191640 1 --2- 172.21.15.161:0/3684663180 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd4840b3470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009d90 tx=0x7fd47c02bea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b949bff64c8f6c01 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.570+0000 7fd48898e640 1 -- 172.21.15.161:0/3684663180 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c02cc50 con 0x7fd48406eca0 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.570+0000 7fd48898e640 1 -- 172.21.15.161:0/3684663180 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd47c02cdb0 con 0x7fd48406eca0 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.570+0000 7fd48898e640 1 -- 172.21.15.161:0/3684663180 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd47c034660 con 0x7fd48406eca0 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.570+0000 7fd48bbf4640 1 -- 172.21.15.161:0/3684663180 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 msgr2=0x7fd4840b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.570+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/3684663180 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd4840b3470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd47c009d90 tx=0x7fd47c02bea0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.571+0000 7fd48bbf4640 1 -- 172.21.15.161:0/3684663180 shutdown_connections 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.571+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/3684663180 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd4840b3470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.571+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/3684663180 >> v2:172.21.15.183:3300/0 conn(0x7fd484072100 0x7fd48415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.571+0000 7fd48bbf4640 1 -- 172.21.15.161:0/3684663180 >> 172.21.15.161:0/3684663180 conn(0x7fd4840b17f0 msgr2=0x7fd4840b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.571+0000 7fd48bbf4640 1 -- 172.21.15.161:0/3684663180 shutdown_connections 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.571+0000 7fd48bbf4640 1 -- 172.21.15.161:0/3684663180 wait complete. 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.572+0000 7fd48bbf4640 1 Processor -- start 2024-03-23T14:36:46.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.572+0000 7fd48bbf4640 1 -- start start 2024-03-23T14:36:46.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.572+0000 7fd48bbf4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd48414e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.572+0000 7fd48bbf4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd484072100 0x7fd4841e34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48bbf4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4840724e0 0x7fd4841e59a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48bbf4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd4841e4800 con 0x7fd484072100 2024-03-23T14:36:46.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48bbf4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd4840cf630 con 0x7fd4840724e0 2024-03-23T14:36:46.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48bbf4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd4841e3bf0 con 0x7fd48406eca0 2024-03-23T14:36:46.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd484072100 0x7fd4841e34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd484072100 0x7fd4841e34f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42094/0 (socket says 172.21.15.161:42094) 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 -- 172.21.15.161:0/1606366073 learned_addr learned my addr 172.21.15.161:0/1606366073 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd489990640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd48414e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48a191640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4840724e0 0x7fd4841e59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 -- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 msgr2=0x7fd48414e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd48414e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 -- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4840724e0 msgr2=0x7fd4841e59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4840724e0 0x7fd4841e59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 -- 172.21.15.161:0/1606366073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd480009400 con 0x7fd484072100 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48a191640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4840724e0 0x7fd4841e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd489990640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd48414e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48918f640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd484072100 0x7fd4841e34f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd47400c410 tx=0x7fd47400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd47a7fc640 1 -- 172.21.15.161:0/1606366073 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4740074e0 con 0x7fd484072100 2024-03-23T14:36:46.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd47a7fc640 1 -- 172.21.15.161:0/1606366073 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd474007660 con 0x7fd484072100 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd47c009a70 con 0x7fd484072100 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd47a7fc640 1 -- 172.21.15.161:0/1606366073 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd474004a20 con 0x7fd484072100 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.573+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd4841e41b0 con 0x7fd484072100 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.574+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd4840735e0 con 0x7fd484072100 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.575+0000 7fd47a7fc640 1 -- 172.21.15.161:0/1606366073 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd474004b80 con 0x7fd484072100 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.575+0000 7fd47a7fc640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd46806b840 0x7fd46806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.575+0000 7fd47a7fc640 1 -- 172.21.15.161:0/1606366073 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7fd474089fa0 con 0x7fd484072100 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.575+0000 7fd489990640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd46806b840 0x7fd46806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.576+0000 7fd489990640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd46806b840 0x7fd46806dd00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd48414ed70 tx=0x7fd480009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.579+0000 7fd47a7fc640 1 -- 172.21.15.161:0/1606366073 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd4740bde10 con 0x7fd484072100 2024-03-23T14:36:46.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.752+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd48404f550 con 0x7fd484072100 2024-03-23T14:36:46.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.753+0000 7fd47a7fc640 1 -- 172.21.15.161:0/1606366073 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v36) ==== 74+0+18517 (secure 0 0 0) 0x7fd4740584a0 con 0x7fd484072100 2024-03-23T14:36:46.749 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:46.749 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":36,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:21.020927+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:04.688180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:05.241774+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"data_new","create_time":"2024-03-23T14:36:18.738957+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"36","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:46.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd46806b840 msgr2=0x7fd46806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd46806b840 0x7fd46806dd00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fd48414ed70 tx=0x7fd480009190 comp rx=0 tx=0).stop 2024-03-23T14:36:46.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd484072100 msgr2=0x7fd4841e34f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd484072100 0x7fd4841e34f0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7fd47400c410 tx=0x7fd47400c8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 shutdown_connections 2024-03-23T14:36:46.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd46806b840 0x7fd46806dd00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd4840724e0 0x7fd4841e59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd484072100 0x7fd4841e34f0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 --2- 172.21.15.161:0/1606366073 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd48406eca0 0x7fd48414e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 >> 172.21.15.161:0/1606366073 conn(0x7fd4840b17f0 msgr2=0x7fd4840b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:46.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 shutdown_connections 2024-03-23T14:36:46.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.755+0000 7fd48bbf4640 1 -- 172.21.15.161:0/1606366073 wait complete. 2024-03-23T14:36:46.762 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T14:36:46.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.867+0000 7f8def1a1640 1 Processor -- start 2024-03-23T14:36:46.863 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.867+0000 7f8def1a1640 1 -- start start 2024-03-23T14:36:46.864 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8def1a1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de80724e0 0x7f8de806e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8def1a1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8de806ead0 0x7f8de80b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8def1a1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8de815be70 con 0x7f8de8072100 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8def1a1640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f8de806ef80 con 0x7f8de806ead0 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8def1a1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8de815c430 con 0x7f8de80724e0 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8ded73e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8de806ead0 0x7f8de80b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8de7fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de80724e0 0x7f8de806e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8decf3d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f8de8072100 0x7f8de815b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:34034/0 (socket says 172.21.15.161:34034) 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8de7fff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de80724e0 0x7f8de806e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:54354/0 (socket says 172.21.15.161:54354) 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8decf3d640 1 -- 172.21.15.161:0/4151571617 learned_addr learned my addr 172.21.15.161:0/4151571617 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.868+0000 7f8ded73e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8de806ead0 0x7f8de80b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:36202/0 (socket says 172.21.15.161:36202) 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.869+0000 7f8ded73e640 1 -- 172.21.15.161:0/4151571617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de80724e0 msgr2=0x7f8de806e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.869+0000 7f8ded73e640 1 --2- 172.21.15.161:0/4151571617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de80724e0 0x7f8de806e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.865 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.869+0000 7f8ded73e640 1 -- 172.21.15.161:0/4151571617 >> v1:172.21.15.183:6789/0 conn(0x7f8de8072100 legacy=0x7f8de815b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.869+0000 7f8ded73e640 1 -- 172.21.15.161:0/4151571617 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de815c150 con 0x7f8de806ead0 2024-03-23T14:36:46.866 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.869+0000 7f8de7fff640 1 --2- 172.21.15.161:0/4151571617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de80724e0 0x7f8de806e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:46.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.869+0000 7f8ded73e640 1 --2- 172.21.15.161:0/4151571617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de806ead0 0x7f8de80b3470 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009d90 tx=0x7f8de002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c926756ed352c2c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8de77fe640 1 -- 172.21.15.161:0/4151571617 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de002cc50 con 0x7f8de806ead0 2024-03-23T14:36:46.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8de77fe640 1 -- 172.21.15.161:0/4151571617 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de002cdb0 con 0x7f8de806ead0 2024-03-23T14:36:46.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8de77fe640 1 -- 172.21.15.161:0/4151571617 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0034660 con 0x7f8de806ead0 2024-03-23T14:36:46.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8def1a1640 1 -- 172.21.15.161:0/4151571617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de806ead0 msgr2=0x7f8de80b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8def1a1640 1 --2- 172.21.15.161:0/4151571617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de806ead0 0x7f8de80b3470 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009d90 tx=0x7f8de002bea0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8def1a1640 1 -- 172.21.15.161:0/4151571617 shutdown_connections 2024-03-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8def1a1640 1 --2- 172.21.15.161:0/4151571617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de806ead0 0x7f8de80b3470 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8def1a1640 1 --2- 172.21.15.161:0/4151571617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de80724e0 0x7f8de806e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8def1a1640 1 -- 172.21.15.161:0/4151571617 >> 172.21.15.161:0/4151571617 conn(0x7f8de80b17f0 msgr2=0x7f8de80b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8def1a1640 1 -- 172.21.15.161:0/4151571617 shutdown_connections 2024-03-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.870+0000 7f8def1a1640 1 -- 172.21.15.161:0/4151571617 wait complete. 2024-03-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.871+0000 7f8def1a1640 1 Processor -- start 2024-03-23T14:36:46.868 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.871+0000 7f8def1a1640 1 -- start start 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8def1a1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8de806ead0 0x7f8de8150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8def1a1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de8072100 0x7f8de813e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8def1a1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de80724e0 0x7f8de813e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8def1a1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8de81e34f0 con 0x7f8de806ead0 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8def1a1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8de814fcd0 con 0x7f8de80724e0 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8def1a1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8de814ffb0 con 0x7f8de8072100 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8decf3d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8de806ead0 0x7f8de8150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8decf3d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8de806ead0 0x7f8de8150b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42106/0 (socket says 172.21.15.161:42106) 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8decf3d640 1 -- 172.21.15.161:0/2602269615 learned_addr learned my addr 172.21.15.161:0/2602269615 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8de7fff640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de8072100 0x7f8de813e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.871 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.872+0000 7f8ded73e640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de80724e0 0x7f8de813e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8decf3d640 1 -- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de8072100 msgr2=0x7f8de813e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8decf3d640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de8072100 0x7f8de813e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8decf3d640 1 -- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de80724e0 msgr2=0x7f8de813e620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8decf3d640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de80724e0 0x7f8de813e620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8decf3d640 1 -- 172.21.15.161:0/2602269615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ddc009400 con 0x7f8de806ead0 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8de7fff640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de8072100 0x7f8de813e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8ded73e640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de80724e0 0x7f8de813e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8decf3d640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8de806ead0 0x7f8de8150b70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800c400 tx=0x7f8dd800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.872 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8de57fa640 1 -- 172.21.15.161:0/2602269615 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd8005ce0 con 0x7f8de806ead0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8de57fa640 1 -- 172.21.15.161:0/2602269615 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8dd8005e40 con 0x7f8de806ead0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8de0009a70 con 0x7f8de806ead0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8de57fa640 1 -- 172.21.15.161:0/2602269615 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dd80072d0 con 0x7f8de806ead0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.873+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8de81e39e0 con 0x7f8de806ead0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.874+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8de8150170 con 0x7f8de806ead0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.874+0000 7f8de57fa640 1 -- 172.21.15.161:0/2602269615 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8dd80074f0 con 0x7f8de806ead0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.875+0000 7f8de57fa640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.875+0000 7f8de57fa640 1 -- 172.21.15.161:0/2602269615 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(36..36 src has 1..36) ==== 6104+0+0 (secure 0 0 0) 0x7f8dd80898d0 con 0x7f8de806ead0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.875+0000 7f8de7fff640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:46.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.875+0000 7f8de7fff640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009fd0 tx=0x7f8ddc009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:46.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:46.879+0000 7f8de57fa640 1 -- 172.21.15.161:0/2602269615 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8dd8054210 con 0x7f8de806ead0 2024-03-23T14:36:47.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.094+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f8de80735e0 con 0x7f8de806ead0 2024-03-23T14:36:47.092 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-23T14:36:47.097+0000 7f8d40811640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T14:36:47.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.099+0000 7f8de57fa640 1 -- 172.21.15.161:0/2602269615 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7f8dd8017090 con 0x7f8de806ead0 2024-03-23T14:36:47.095 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T14:36:47.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 msgr2=0x7f8dd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:47.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f8ddc009fd0 tx=0x7f8ddc009190 comp rx=0 tx=0).stop 2024-03-23T14:36:47.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8de806ead0 msgr2=0x7f8de8150b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:47.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8de806ead0 0x7f8de8150b70 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8dd800c400 tx=0x7f8dd800c8c0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 shutdown_connections 2024-03-23T14:36:47.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8de80724e0 0x7f8de813e620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8de8072100 0x7f8de813e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 --2- 172.21.15.161:0/2602269615 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8de806ead0 0x7f8de8150b70 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.102+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 >> 172.21.15.161:0/2602269615 conn(0x7f8de80b17f0 msgr2=0x7f8de8155230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:47.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.103+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 shutdown_connections 2024-03-23T14:36:47.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.103+0000 7f8def1a1640 1 -- 172.21.15.161:0/2602269615 wait complete. 2024-03-23T14:36:47.100 INFO:tasks.ceph.mds.d.smithi183.stderr: -3607> 2024-03-23T14:36:11.911+0000 7f579843b640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T14:36:47.110 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T14:36:47.127 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-23T14:36:47.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.217+0000 7fe4c75d0640 1 Processor -- start 2024-03-23T14:36:47.213 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.217+0000 7fe4c75d0640 1 -- start start 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4c75d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4c0161c40 0x7fe4c01620c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4c75d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c00adfc0 con 0x7fe4c0146d60 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4c75d0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4c0162700 con 0x7fe4c00721f0 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4c75d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c01629e0 con 0x7fe4c0161c40 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4c536c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe4c0146d60 0x7fe4c00ad8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35242/0 (socket says 172.21.15.161:35242) 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4c536c640 1 -- 172.21.15.161:0/539904099 learned_addr learned my addr 172.21.15.161:0/539904099 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1556339037 0 0) 0x7fe4c00adfc0 con 0x7fe4c0146d60 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4ac003520 con 0x7fe4c0146d60 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4c4b6b640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4c0161c40 0x7fe4c01620c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3143914906 0 0) 0x7fe4ac003520 con 0x7fe4c0146d60 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4c0161c40 msgr2=0x7fe4c01620c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:47.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4c0161c40 0x7fe4c01620c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 >> v1:172.21.15.183:6789/0 conn(0x7fe4c00721f0 legacy=0x7fe4c00b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c00b43c0 con 0x7fe4c0146d60 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fe4bc001670 con 0x7fe4c0146d60 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 >> v1:172.21.15.161:6789/0 conn(0x7fe4c0146d60 legacy=0x7fe4c00ad8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.218+0000 7fe4b7fff640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4ac0039a0 0x7fe4ac003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4b7fff640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4c0161c40 0x7fe4ac0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4b7fff640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4ac004d00 0x7fe4ac005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c01629e0 con 0x7fe4ac0039a0 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe4ac00ba30 con 0x7fe4ac004d00 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe4ac00bbf0 con 0x7fe4c0161c40 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c536c640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4ac0039a0 0x7fe4ac003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c4b6b640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4c0161c40 0x7fe4ac0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c5b6d640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4ac004d00 0x7fe4ac005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:47.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c536c640 1 -- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4c0161c40 msgr2=0x7fe4ac0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c536c640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4c0161c40 0x7fe4ac0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c536c640 1 -- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4ac004d00 msgr2=0x7fe4ac005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c536c640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4ac004d00 0x7fe4ac005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c536c640 1 -- 172.21.15.161:0/539904099 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4ac003520 con 0x7fe4ac0039a0 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.219+0000 7fe4c5b6d640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4ac004d00 0x7fe4ac005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4c536c640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4ac0039a0 0x7fe4ac003dc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00c210 tx=0x7fe4ac0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18bf58bb91b7f3bc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc005ad0 con 0x7fe4ac0039a0 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4bc005c30 con 0x7fe4ac0039a0 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4b7fff640 1 -- 172.21.15.161:0/539904099 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc02c890 con 0x7fe4ac0039a0 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4c75d0640 1 -- 172.21.15.161:0/539904099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4ac0039a0 msgr2=0x7fe4ac003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4c75d0640 1 --2- 172.21.15.161:0/539904099 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4ac0039a0 0x7fe4ac003dc0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc00c210 tx=0x7fe4ac0057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4c5b6d640 1 -- 172.21.15.161:0/539904099 reap_dead start 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4c75d0640 1 -- 172.21.15.161:0/539904099 shutdown_connections 2024-03-23T14:36:47.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4c75d0640 1 -- 172.21.15.161:0/539904099 >> 172.21.15.161:0/539904099 conn(0x7fe4c006ce50 msgr2=0x7fe4c006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:47.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4c75d0640 1 -- 172.21.15.161:0/539904099 shutdown_connections 2024-03-23T14:36:47.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.220+0000 7fe4c75d0640 1 -- 172.21.15.161:0/539904099 wait complete. 2024-03-23T14:36:47.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c75d0640 1 Processor -- start 2024-03-23T14:36:47.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c75d0640 1 -- start start 2024-03-23T14:36:47.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c75d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4ac0039a0 0x7fe4c0160f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c75d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c01f4400 0x7fe4c01399a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c75d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4c0139ee0 0x7fe4c00ce560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c75d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c01f53a0 con 0x7fe4c01f4400 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c75d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe4c0161490 con 0x7fe4c0139ee0 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c75d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe4c01f4de0 con 0x7fe4ac0039a0 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c4b6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c01f4400 0x7fe4c01399a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c4b6b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c01f4400 0x7fe4c01399a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42118/0 (socket says 172.21.15.161:42118) 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.222+0000 7fe4c4b6b640 1 -- 172.21.15.161:0/2070311553 learned_addr learned my addr 172.21.15.161:0/2070311553 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c5b6d640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4c0139ee0 0x7fe4c00ce560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c536c640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4ac0039a0 0x7fe4c0160f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c4b6b640 1 -- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4ac0039a0 msgr2=0x7fe4c0160f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c4b6b640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4ac0039a0 0x7fe4c0160f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c4b6b640 1 -- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4c0139ee0 msgr2=0x7fe4c00ce560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:47.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c4b6b640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4c0139ee0 0x7fe4c00ce560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c4b6b640 1 -- 172.21.15.161:0/2070311553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c01f50c0 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c5b6d640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4c0139ee0 0x7fe4c00ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c4b6b640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c01f4400 0x7fe4c01399a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0011960 tx=0x7fe4b0011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4b5ffb640 1 -- 172.21.15.161:0/2070311553 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b000faf0 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4b5ffb640 1 -- 172.21.15.161:0/2070311553 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4b000fc50 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4c01391a0 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4b5ffb640 1 -- 172.21.15.161:0/2070311553 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4b0014440 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.223+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4c013b7b0 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.224+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4c00afaf0 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.225+0000 7fe4b5ffb640 1 -- 172.21.15.161:0/2070311553 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe4b00145a0 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.225+0000 7fe4b5ffb640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4a406b810 0x7fe4a406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.225+0000 7fe4b5ffb640 1 -- 172.21.15.161:0/2070311553 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6190+0+0 (secure 0 0 0) 0x7fe4b008aba0 con 0x7fe4c01f4400 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.226+0000 7fe4c536c640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4a406b810 0x7fe4a406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:47.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.226+0000 7fe4c536c640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4a406b810 0x7fe4a406dcd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc034000 tx=0x7fe4bc009030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:47.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.229+0000 7fe4b5ffb640 1 -- 172.21.15.161:0/2070311553 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe4b0055470 con 0x7fe4c01f4400 2024-03-23T14:36:47.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:47.437+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fe4c00b3130 con 0x7fe4c01f4400 2024-03-23T14:36:48.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.102+0000 7fe4b5ffb640 1 -- 172.21.15.161:0/2070311553 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7fe4b0058b30 con 0x7fe4c01f4400 2024-03-23T14:36:48.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4a406b810 msgr2=0x7fe4a406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4a406b810 0x7fe4a406dcd0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc034000 tx=0x7fe4bc009030 comp rx=0 tx=0).stop 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c01f4400 msgr2=0x7fe4c01399a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c01f4400 0x7fe4c01399a0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7fe4b0011960 tx=0x7fe4b0011e20 comp rx=0 tx=0).stop 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 shutdown_connections 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4a406b810 0x7fe4a406dcd0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4c0139ee0 0x7fe4c00ce560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4c01f4400 0x7fe4c01399a0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 --2- 172.21.15.161:0/2070311553 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4ac0039a0 0x7fe4c0160f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 >> 172.21.15.161:0/2070311553 conn(0x7fe4c006ce50 msgr2=0x7fe4c0066e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.104+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 shutdown_connections 2024-03-23T14:36:48.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.105+0000 7fe4c75d0640 1 -- 172.21.15.161:0/2070311553 wait complete. 2024-03-23T14:36:48.113 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T14:36:48.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.226+0000 7fc749ab1640 1 Processor -- start 2024-03-23T14:36:48.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.226+0000 7fc749ab1640 1 -- start start 2024-03-23T14:36:48.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc749ab1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc744145720 0x7fc744160310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:48.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc749ab1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc7440cfde0 0x7fc74415c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:48.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc749ab1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc744145b00 con 0x7fc74415c890 2024-03-23T14:36:48.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc749ab1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc744160850 con 0x7fc744145720 2024-03-23T14:36:48.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc749ab1640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc74415cdd0 con 0x7fc7440cfde0 2024-03-23T14:36:48.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc7437fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc744145720 0x7fc744160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:48.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc7437fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc744145720 0x7fc744160310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42134/0 (socket says 172.21.15.161:42134) 2024-03-23T14:36:48.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc7437fe640 1 -- 172.21.15.161:0/182361481 learned_addr learned my addr 172.21.15.161:0/182361481 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:48.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc742ffd640 1 --2- 172.21.15.161:0/182361481 >> v2:172.21.15.183:3300/0 conn(0x7fc7440cfde0 0x7fc74415c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:48.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.227+0000 7fc7437fe640 1 -- 172.21.15.161:0/182361481 >> v2:172.21.15.183:3300/0 conn(0x7fc7440cfde0 msgr2=0x7fc74415c350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:48.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.228+0000 7fc7437fe640 1 --2- 172.21.15.161:0/182361481 >> v2:172.21.15.183:3300/0 conn(0x7fc7440cfde0 0x7fc74415c350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.228+0000 7fc7437fe640 1 -- 172.21.15.161:0/182361481 >> v1:172.21.15.183:6789/0 conn(0x7fc74415c890 legacy=0x7fc7440ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:48.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.228+0000 7fc7437fe640 1 -- 172.21.15.161:0/182361481 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc74415d390 con 0x7fc744145720 2024-03-23T14:36:48.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.228+0000 7fc7437fe640 1 --2- 172.21.15.161:0/182361481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc744145720 0x7fc744160310 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc7440cdea0 tx=0x7fc73402b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa5daa26ad279e92 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:48.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.228+0000 7fc7427fc640 1 -- 172.21.15.161:0/182361481 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734030860 con 0x7fc744145720 2024-03-23T14:36:48.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.228+0000 7fc7427fc640 1 -- 172.21.15.161:0/182361481 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc7340309c0 con 0x7fc744145720 2024-03-23T14:36:48.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.228+0000 7fc7427fc640 1 -- 172.21.15.161:0/182361481 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734030d50 con 0x7fc744145720 2024-03-23T14:36:48.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.229+0000 7fc749ab1640 1 -- 172.21.15.161:0/182361481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc744145720 msgr2=0x7fc744160310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:48.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.229+0000 7fc749ab1640 1 --2- 172.21.15.161:0/182361481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc744145720 0x7fc744160310 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7fc7440cdea0 tx=0x7fc73402b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.229+0000 7fc749ab1640 1 -- 172.21.15.161:0/182361481 shutdown_connections 2024-03-23T14:36:48.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.229+0000 7fc749ab1640 1 --2- 172.21.15.161:0/182361481 >> v2:172.21.15.183:3300/0 conn(0x7fc7440cfde0 0x7fc74415c350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.229+0000 7fc749ab1640 1 --2- 172.21.15.161:0/182361481 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc744145720 0x7fc744160310 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.229+0000 7fc749ab1640 1 -- 172.21.15.161:0/182361481 >> 172.21.15.161:0/182361481 conn(0x7fc74406b260 msgr2=0x7fc74406b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:48.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.229+0000 7fc749ab1640 1 -- 172.21.15.161:0/182361481 shutdown_connections 2024-03-23T14:36:48.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.229+0000 7fc749ab1640 1 -- 172.21.15.161:0/182361481 wait complete. 2024-03-23T14:36:48.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.230+0000 7fc749ab1640 1 Processor -- start 2024-03-23T14:36:48.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.230+0000 7fc749ab1640 1 -- start start 2024-03-23T14:36:48.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc749ab1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7440cfde0 0x7fc74406d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:48.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc749ab1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc744145720 0x7fc74406db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:48.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc749ab1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc74415c890 0x7fc74406e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:48.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc749ab1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc7440ae880 con 0x7fc7440cfde0 2024-03-23T14:36:48.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc749ab1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc7440aeb60 con 0x7fc744145720 2024-03-23T14:36:48.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc749ab1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc7440aee40 con 0x7fc74415c890 2024-03-23T14:36:48.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc7437fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7440cfde0 0x7fc74406d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:48.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc7437fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7440cfde0 0x7fc74406d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42136/0 (socket says 172.21.15.161:42136) 2024-03-23T14:36:48.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc7437fe640 1 -- 172.21.15.161:0/205638284 learned_addr learned my addr 172.21.15.161:0/205638284 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc743fff640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc74415c890 0x7fc74406e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.231+0000 7fc742ffd640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc744145720 0x7fc74406db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc7437fe640 1 -- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc74415c890 msgr2=0x7fc74406e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:48.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc7437fe640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc74415c890 0x7fc74406e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc7437fe640 1 -- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc744145720 msgr2=0x7fc74406db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:48.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc7437fe640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc744145720 0x7fc74406db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:48.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc7437fe640 1 -- 172.21.15.161:0/205638284 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7340094d0 con 0x7fc7440cfde0 2024-03-23T14:36:48.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc743fff640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc74415c890 0x7fc74406e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:48.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc742ffd640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc744145720 0x7fc74406db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:48.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc7437fe640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7440cfde0 0x7fc74406d620 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc734000c00 tx=0x7fc734004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:48.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc72bfff640 1 -- 172.21.15.161:0/205638284 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734039c90 con 0x7fc7440cfde0 2024-03-23T14:36:48.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc72bfff640 1 -- 172.21.15.161:0/205638284 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc734039df0 con 0x7fc7440cfde0 2024-03-23T14:36:48.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc72bfff640 1 -- 172.21.15.161:0/205638284 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc734043620 con 0x7fc7440cfde0 2024-03-23T14:36:48.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7440af120 con 0x7fc7440cfde0 2024-03-23T14:36:48.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.232+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7441386e0 con 0x7fc7440cfde0 2024-03-23T14:36:48.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.233+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc74406d340 con 0x7fc7440cfde0 2024-03-23T14:36:48.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.233+0000 7fc72bfff640 1 -- 172.21.15.161:0/205638284 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc734043780 con 0x7fc7440cfde0 2024-03-23T14:36:48.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.234+0000 7fc72bfff640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc73006b810 0x7fc73006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:48.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.234+0000 7fc72bfff640 1 -- 172.21.15.161:0/205638284 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(37..37 src has 1..37) ==== 6190+0+0 (secure 0 0 0) 0x7fc73403e070 con 0x7fc7440cfde0 2024-03-23T14:36:48.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.234+0000 7fc742ffd640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc73006b810 0x7fc73006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:48.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.234+0000 7fc742ffd640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc73006b810 0x7fc73006dcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc744160680 tx=0x7fc738008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:48.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.238+0000 7fc72bfff640 1 -- 172.21.15.161:0/205638284 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc74406d340 con 0x7fc7440cfde0 2024-03-23T14:36:48.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:48.412+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc74404f4e0 con 0x7fc7440cfde0 2024-03-23T14:36:49.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.108+0000 7fc72bfff640 1 -- 172.21.15.161:0/205638284 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v38) ==== 181+0+0 (secure 0 0 0) 0x7fc73407b840 con 0x7fc7440cfde0 2024-03-23T14:36:49.104 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-23T14:36:49.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc73006b810 msgr2=0x7fc73006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:49.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc73006b810 0x7fc73006dcd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fc744160680 tx=0x7fc738008040 comp rx=0 tx=0).stop 2024-03-23T14:36:49.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7440cfde0 msgr2=0x7fc74406d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:49.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7440cfde0 0x7fc74406d620 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7fc734000c00 tx=0x7fc734004730 comp rx=0 tx=0).stop 2024-03-23T14:36:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 shutdown_connections 2024-03-23T14:36:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc73006b810 0x7fc73006dcd0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc74415c890 0x7fc74406e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc744145720 0x7fc74406db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 --2- 172.21.15.161:0/205638284 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc7440cfde0 0x7fc74406d620 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 >> 172.21.15.161:0/205638284 conn(0x7fc74406b260 msgr2=0x7fc744143d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 shutdown_connections 2024-03-23T14:36:49.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.110+0000 7fc749ab1640 1 -- 172.21.15.161:0/205638284 wait complete. 2024-03-23T14:36:49.121 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T14:36:49.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.232+0000 7fa9309a0640 1 Processor -- start 2024-03-23T14:36:49.228 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.232+0000 7fa9309a0640 1 -- start start 2024-03-23T14:36:49.229 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa9309a0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa92c072b40 0x7fa92c06fcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa9309a0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa92c06dc00 con 0x7fa92c06e090 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa9309a0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa92c071ed0 con 0x7fa92c15dff0 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa9309a0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa92c070230 con 0x7fa92c072b40 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa92affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa92c072b40 0x7fa92c06fcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa92bfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa92c06e090 0x7fa92c06e4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35248/0 (socket says 172.21.15.161:35248) 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa92affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa92c072b40 0x7fa92c06fcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42138/0 (socket says 172.21.15.161:42138) 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa92bfff640 1 -- 172.21.15.161:0/3696467808 learned_addr learned my addr 172.21.15.161:0/3696467808 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa92affd640 1 -- 172.21.15.161:0/3696467808 >> v1:172.21.15.183:6789/0 conn(0x7fa92c15dff0 legacy=0x7fa92c072430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa92affd640 1 -- 172.21.15.161:0/3696467808 >> v1:172.21.15.161:6789/0 conn(0x7fa92c06e090 legacy=0x7fa92c06e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa92a7fc640 1 -- 172.21.15.161:0/3696467808 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2902709538 0 0) 0x7fa92c06dc00 con 0x7fa92c06e090 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.233+0000 7fa92affd640 1 -- 172.21.15.161:0/3696467808 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa92c06d8d0 con 0x7fa92c072b40 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.234+0000 7fa92affd640 1 --2- 172.21.15.161:0/3696467808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c072b40 0x7fa92c06fcf0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa9200097a0 tx=0x7fa92002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c1f63893126eae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:49.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.234+0000 7fa92a7fc640 1 -- 172.21.15.161:0/3696467808 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920031830 con 0x7fa92c072b40 2024-03-23T14:36:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.234+0000 7fa92a7fc640 1 -- 172.21.15.161:0/3696467808 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa920031990 con 0x7fa92c072b40 2024-03-23T14:36:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.234+0000 7fa92a7fc640 1 -- 172.21.15.161:0/3696467808 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa920031d20 con 0x7fa92c072b40 2024-03-23T14:36:49.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.234+0000 7fa9309a0640 1 -- 172.21.15.161:0/3696467808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c072b40 msgr2=0x7fa92c06fcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.234+0000 7fa9309a0640 1 --2- 172.21.15.161:0/3696467808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c072b40 0x7fa92c06fcf0 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7fa9200097a0 tx=0x7fa92002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.235+0000 7fa9309a0640 1 -- 172.21.15.161:0/3696467808 shutdown_connections 2024-03-23T14:36:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.235+0000 7fa9309a0640 1 --2- 172.21.15.161:0/3696467808 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c072b40 0x7fa92c06fcf0 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.235+0000 7fa9309a0640 1 -- 172.21.15.161:0/3696467808 >> 172.21.15.161:0/3696467808 conn(0x7fa92c0b2720 msgr2=0x7fa92c0b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.235+0000 7fa9309a0640 1 -- 172.21.15.161:0/3696467808 shutdown_connections 2024-03-23T14:36:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.235+0000 7fa9309a0640 1 -- 172.21.15.161:0/3696467808 wait complete. 2024-03-23T14:36:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.236+0000 7fa9309a0640 1 Processor -- start 2024-03-23T14:36:49.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.236+0000 7fa9309a0640 1 -- start start 2024-03-23T14:36:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa9309a0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa92c06e090 0x7fa92c15f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa9309a0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa92c072b40 0x7fa92c160e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa9309a0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c15dff0 0x7fa92c1f3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa9309a0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa92c161380 con 0x7fa92c15dff0 2024-03-23T14:36:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa9309a0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa92c1f3650 con 0x7fa92c06e090 2024-03-23T14:36:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa9309a0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa92c148d20 con 0x7fa92c072b40 2024-03-23T14:36:49.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c15dff0 0x7fa92c1f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c15dff0 0x7fa92c1f3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42150/0 (socket says 172.21.15.161:42150) 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 -- 172.21.15.161:0/2201142687 learned_addr learned my addr 172.21.15.161:0/2201142687 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92b7fe640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa92c06e090 0x7fa92c15f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92affd640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa92c072b40 0x7fa92c160e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 -- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa92c072b40 msgr2=0x7fa92c160e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa92c072b40 0x7fa92c160e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 -- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa92c06e090 msgr2=0x7fa92c15f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa92c06e090 0x7fa92c15f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 -- 172.21.15.161:0/2201142687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa920009400 con 0x7fa92c15dff0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa92bfff640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c15dff0 0x7fa92c1f3110 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa92400c7e0 tx=0x7fa92400cca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa913fff640 1 -- 172.21.15.161:0/2201142687 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92400a640 con 0x7fa92c15dff0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa913fff640 1 -- 172.21.15.161:0/2201142687 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa92400a7c0 con 0x7fa92c15dff0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa92c15f710 con 0x7fa92c15dff0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa913fff640 1 -- 172.21.15.161:0/2201142687 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa92400ab70 con 0x7fa92c15dff0 2024-03-23T14:36:49.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.237+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa92c15fcd0 con 0x7fa92c15dff0 2024-03-23T14:36:49.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.238+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa92c068680 con 0x7fa92c15dff0 2024-03-23T14:36:49.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.239+0000 7fa913fff640 1 -- 172.21.15.161:0/2201142687 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa92400acd0 con 0x7fa92c15dff0 2024-03-23T14:36:49.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.239+0000 7fa913fff640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa91806b810 0x7fa91806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:49.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.239+0000 7fa913fff640 1 -- 172.21.15.161:0/2201142687 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 5752+0+0 (secure 0 0 0) 0x7fa924089980 con 0x7fa92c15dff0 2024-03-23T14:36:49.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.239+0000 7fa92b7fe640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa91806b810 0x7fa91806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:49.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.240+0000 7fa92b7fe640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa91806b810 0x7fa91806dcd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa92c15d040 tx=0x7fa91c009060 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:49.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.243+0000 7fa913fff640 1 -- 172.21.15.161:0/2201142687 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa9240544b0 con 0x7fa92c15dff0 2024-03-23T14:36:49.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:49.416+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fa92c04f470 con 0x7fa92c15dff0 2024-03-23T14:36:50.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.109+0000 7fa913fff640 1 -- 172.21.15.161:0/2201142687 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v39) ==== 169+0+0 (secure 0 0 0) 0x7fa924018020 con 0x7fa92c15dff0 2024-03-23T14:36:50.105 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa91806b810 msgr2=0x7fa91806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa91806b810 0x7fa91806dcd0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa92c15d040 tx=0x7fa91c009060 comp rx=0 tx=0).stop 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c15dff0 msgr2=0x7fa92c1f3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c15dff0 0x7fa92c1f3110 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fa92400c7e0 tx=0x7fa92400cca0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 shutdown_connections 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa91806b810 0x7fa91806dcd0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa92c15dff0 0x7fa92c1f3110 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa92c072b40 0x7fa92c160e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 --2- 172.21.15.161:0/2201142687 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa92c06e090 0x7fa92c15f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 >> 172.21.15.161:0/2201142687 conn(0x7fa92c0b2720 msgr2=0x7fa92c161cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:50.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 shutdown_connections 2024-03-23T14:36:50.109 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.112+0000 7fa9309a0640 1 -- 172.21.15.161:0/2201142687 wait complete. 2024-03-23T14:36:50.126 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm data_new data_new --yes-i-really-really-mean-it 2024-03-23T14:36:50.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.237+0000 7f25c84f4640 1 Processor -- start 2024-03-23T14:36:50.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.237+0000 7f25c84f4640 1 -- start start 2024-03-23T14:36:50.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.238+0000 7f25c84f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 0x7f25c00d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:50.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.238+0000 7f25c84f4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f25c015e7a0 0x7f25c00d2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:50.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.238+0000 7f25c84f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25c00d0a30 con 0x7f25c00b3a80 2024-03-23T14:36:50.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.238+0000 7f25c84f4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f25c00d0d10 con 0x7f25c015e7a0 2024-03-23T14:36:50.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.238+0000 7f25c84f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f25c00d0ff0 con 0x7f25c015e3c0 2024-03-23T14:36:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.239+0000 7f25c5a8f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f25c015e7a0 0x7f25c00d2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.239+0000 7f25c6290640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 0x7f25c00d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.239+0000 7f25c6a91640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f25c00b3a80 0x7f25c00b3f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:34070/0 (socket says 172.21.15.161:34070) 2024-03-23T14:36:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.239+0000 7f25c6a91640 1 -- 172.21.15.161:0/2961547709 learned_addr learned my addr 172.21.15.161:0/2961547709 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.239+0000 7f25c5a8f640 1 -- 172.21.15.161:0/2961547709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 msgr2=0x7f25c00d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.239+0000 7f25c5a8f640 1 --2- 172.21.15.161:0/2961547709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 0x7f25c00d23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.239+0000 7f25c5a8f640 1 -- 172.21.15.161:0/2961547709 >> v1:172.21.15.183:6789/0 conn(0x7f25c00b3a80 legacy=0x7f25c00b3f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:50.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.239+0000 7f25c5a8f640 1 -- 172.21.15.161:0/2961547709 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25c00d12d0 con 0x7f25c015e7a0 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c5a8f640 1 --2- 172.21.15.161:0/2961547709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 0x7f25c00d2920 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f25b00097a0 tx=0x7f25b002b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e1d664b9fda91a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c528e640 1 -- 172.21.15.161:0/2961547709 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0030610 con 0x7f25c015e7a0 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c528e640 1 -- 172.21.15.161:0/2961547709 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25b0030770 con 0x7f25c015e7a0 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c528e640 1 -- 172.21.15.161:0/2961547709 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25b0030b00 con 0x7f25c015e7a0 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c84f4640 1 -- 172.21.15.161:0/2961547709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 msgr2=0x7f25c00d2920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c84f4640 1 --2- 172.21.15.161:0/2961547709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 0x7f25c00d2920 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f25b00097a0 tx=0x7f25b002b960 comp rx=0 tx=0).stop 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c84f4640 1 -- 172.21.15.161:0/2961547709 shutdown_connections 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c84f4640 1 --2- 172.21.15.161:0/2961547709 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 0x7f25c00d2920 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c84f4640 1 --2- 172.21.15.161:0/2961547709 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 0x7f25c00d23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c84f4640 1 -- 172.21.15.161:0/2961547709 >> 172.21.15.161:0/2961547709 conn(0x7f25c009d1d0 msgr2=0x7f25c00d3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c84f4640 1 -- 172.21.15.161:0/2961547709 shutdown_connections 2024-03-23T14:36:50.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.240+0000 7f25c84f4640 1 -- 172.21.15.161:0/2961547709 wait complete. 2024-03-23T14:36:50.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c84f4640 1 Processor -- start 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c84f4640 1 -- start start 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c84f4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25c00b3a80 0x7f25c0152ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c84f4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 0x7f25c0161160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c84f4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 0x7f25c00cc9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c84f4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f25c0160510 con 0x7f25c00b3a80 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c84f4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f25c0153220 con 0x7f25c015e7a0 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c84f4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f25c015f900 con 0x7f25c015e3c0 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c6290640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25c00b3a80 0x7f25c0152ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c6290640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25c00b3a80 0x7f25c0152ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:42162/0 (socket says 172.21.15.161:42162) 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c6290640 1 -- 172.21.15.161:0/1214767617 learned_addr learned my addr 172.21.15.161:0/1214767617 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.242+0000 7f25c6a91640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 0x7f25c00cc9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c5a8f640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 0x7f25c0161160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:50.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c6290640 1 -- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 msgr2=0x7f25c0161160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c6290640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 0x7f25c0161160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c6290640 1 -- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 msgr2=0x7f25c00cc9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c6290640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 0x7f25c00cc9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c6290640 1 -- 172.21.15.161:0/1214767617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25b0009400 con 0x7f25c00b3a80 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c6290640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25c00b3a80 0x7f25c0152ce0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00efc0 tx=0x7f25bc00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25b6ffd640 1 -- 172.21.15.161:0/1214767617 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc014480 con 0x7f25c00b3a80 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25b6ffd640 1 -- 172.21.15.161:0/1214767617 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f25bc00f040 con 0x7f25c00b3a80 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25c015fbe0 con 0x7f25c00b3a80 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25b6ffd640 1 -- 172.21.15.161:0/1214767617 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25bc014800 con 0x7f25c00b3a80 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.243+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25c01601a0 con 0x7f25c00b3a80 2024-03-23T14:36:50.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.244+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25c00c8350 con 0x7f25c00b3a80 2024-03-23T14:36:50.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.244+0000 7f25b6ffd640 1 -- 172.21.15.161:0/1214767617 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f25bc014960 con 0x7f25c00b3a80 2024-03-23T14:36:50.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.245+0000 7f25b6ffd640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f25a406b840 0x7f25a406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:50.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.245+0000 7f25b6ffd640 1 -- 172.21.15.161:0/1214767617 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(39..39 src has 1..39) ==== 5338+0+0 (secure 0 0 0) 0x7f25bc08ab40 con 0x7f25c00b3a80 2024-03-23T14:36:50.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.245+0000 7f25c5a8f640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f25a406b840 0x7f25a406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:50.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.245+0000 7f25c5a8f640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f25a406b840 0x7f25a406dd00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f25b0036000 tx=0x7f25b003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:50.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.249+0000 7f25b6ffd640 1 -- 172.21.15.161:0/1214767617 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f25bc055780 con 0x7f25c00b3a80 2024-03-23T14:36:50.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:50.421+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true} v 0) -- 0x7f25c004f4e0 con 0x7f25c00b3a80 2024-03-23T14:36:51.119 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.123+0000 7f25b6ffd640 1 -- 172.21.15.161:0/1214767617 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "data_new", "pool2": "data_new", "yes_i_really_really_mean_it": true}]=0 pool 'data_new' removed v40) ==== 160+0+0 (secure 0 0 0) 0x7f25bc058e40 con 0x7f25c00b3a80 2024-03-23T14:36:51.392 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'data_new' removed 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f25a406b840 msgr2=0x7f25a406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f25a406b840 0x7f25a406dd00 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f25b0036000 tx=0x7f25b003b040 comp rx=0 tx=0).stop 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25c00b3a80 msgr2=0x7f25c0152ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25c00b3a80 0x7f25c0152ce0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f25bc00efc0 tx=0x7f25bc00c490 comp rx=0 tx=0).stop 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 shutdown_connections 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f25a406b840 0x7f25a406dd00 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f25c015e7a0 0x7f25c00cc9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f25c015e3c0 0x7f25c0161160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 --2- 172.21.15.161:0/1214767617 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f25c00b3a80 0x7f25c0152ce0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 >> 172.21.15.161:0/1214767617 conn(0x7f25c009d1d0 msgr2=0x7f25c00b14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:51.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 shutdown_connections 2024-03-23T14:36:51.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.127+0000 7f25c84f4640 1 -- 172.21.15.161:0/1214767617 wait complete. 2024-03-23T14:36:51.395 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T14:36:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.503+0000 7f511f464640 1 Processor -- start 2024-03-23T14:36:51.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.503+0000 7f511f464640 1 -- start start 2024-03-23T14:36:51.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511f464640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f51180d0a20 0x7f51180d2340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511f464640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51180c5cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511f464640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51180ceb30 con 0x7f511815c1d0 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511f464640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f51180cee10 con 0x7f51180d0a20 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511f464640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f51180cf0f0 con 0x7f51180ce3c0 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511d200640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f511815c1d0 0x7f51180d0310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48434/0 (socket says 172.21.15.161:48434) 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511d200640 1 -- 172.21.15.161:0/2822543595 learned_addr learned my addr 172.21.15.161:0/2822543595 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511c9ff640 1 --2- 172.21.15.161:0/2822543595 >> v2:172.21.15.183:3300/0 conn(0x7f51180d0a20 0x7f51180d2340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.504+0000 7f511da01640 1 --2- 172.21.15.161:0/2822543595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51180c5cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.505+0000 7f510ffff640 1 -- 172.21.15.161:0/2822543595 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 526535022 0 0) 0x7f51180ceb30 con 0x7f511815c1d0 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.505+0000 7f510ffff640 1 -- 172.21.15.161:0/2822543595 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5104003520 con 0x7f511815c1d0 2024-03-23T14:36:51.501 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.505+0000 7f511da01640 1 -- 172.21.15.161:0/2822543595 >> v2:172.21.15.183:3300/0 conn(0x7f51180d0a20 msgr2=0x7f51180d2340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:51.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.505+0000 7f511da01640 1 --2- 172.21.15.161:0/2822543595 >> v2:172.21.15.183:3300/0 conn(0x7f51180d0a20 0x7f51180d2340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.505+0000 7f511da01640 1 -- 172.21.15.161:0/2822543595 >> v1:172.21.15.161:6789/0 conn(0x7f511815c1d0 legacy=0x7f51180d0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.505+0000 7f511da01640 1 -- 172.21.15.161:0/2822543595 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51180cf3d0 con 0x7f51180ce3c0 2024-03-23T14:36:51.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.505+0000 7f511c9ff640 1 --2- 172.21.15.161:0/2822543595 >> v2:172.21.15.183:3300/0 conn(0x7f51180d0a20 0x7f51180d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:51.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.506+0000 7f511da01640 1 --2- 172.21.15.161:0/2822543595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51180c5cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f51100097a0 tx=0x7f511002b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23d716f66ced2d2b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:51.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.506+0000 7f510ffff640 1 -- 172.21.15.161:0/2822543595 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f511002e070 con 0x7f51180ce3c0 2024-03-23T14:36:51.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.506+0000 7f510ffff640 1 -- 172.21.15.161:0/2822543595 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51100315a0 con 0x7f51180ce3c0 2024-03-23T14:36:51.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.506+0000 7f510ffff640 1 -- 172.21.15.161:0/2822543595 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110031910 con 0x7f51180ce3c0 2024-03-23T14:36:51.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.506+0000 7f511f464640 1 -- 172.21.15.161:0/2822543595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 msgr2=0x7f51180c5cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.506+0000 7f511f464640 1 --2- 172.21.15.161:0/2822543595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51180c5cf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f51100097a0 tx=0x7f511002b840 comp rx=0 tx=0).stop 2024-03-23T14:36:51.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.507+0000 7f511f464640 1 -- 172.21.15.161:0/2822543595 shutdown_connections 2024-03-23T14:36:51.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.507+0000 7f511f464640 1 --2- 172.21.15.161:0/2822543595 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51180c5cf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.507+0000 7f511f464640 1 --2- 172.21.15.161:0/2822543595 >> v2:172.21.15.183:3300/0 conn(0x7f51180d0a20 0x7f51180d2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.507+0000 7f511f464640 1 -- 172.21.15.161:0/2822543595 >> 172.21.15.161:0/2822543595 conn(0x7f511809aa40 msgr2=0x7f511809ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:51.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.507+0000 7f511f464640 1 -- 172.21.15.161:0/2822543595 shutdown_connections 2024-03-23T14:36:51.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.507+0000 7f511f464640 1 -- 172.21.15.161:0/2822543595 wait complete. 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.508+0000 7f511f464640 1 Processor -- start 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.508+0000 7f511f464640 1 -- start start 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.508+0000 7f511f464640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51181f3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.508+0000 7f511f464640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51180d0a20 0x7f5118153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.508+0000 7f511f464640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f511815c1d0 0x7f511815e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.508+0000 7f511f464640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f51181f41b0 con 0x7f511815c1d0 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.508+0000 7f511f464640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f511806c190 con 0x7f51180d0a20 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.508+0000 7f511f464640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f511809f8a0 con 0x7f51180ce3c0 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f511815c1d0 0x7f511815e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f511815c1d0 0x7f511815e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44898/0 (socket says 172.21.15.161:44898) 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 -- 172.21.15.161:0/424866407 learned_addr learned my addr 172.21.15.161:0/424866407 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511c9ff640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51180d0a20 0x7f5118153240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511d200640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51181f3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 -- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 msgr2=0x7f51181f3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51181f3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 -- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51180d0a20 msgr2=0x7f5118153240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51180d0a20 0x7f5118153240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 -- 172.21.15.161:0/424866407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5108009400 con 0x7f511815c1d0 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511da01640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f511815c1d0 0x7f511815e780 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5110034040 tx=0x7f51100046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f510dffb640 1 -- 172.21.15.161:0/424866407 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110032630 con 0x7f511815c1d0 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f510dffb640 1 -- 172.21.15.161:0/424866407 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5110032790 con 0x7f511815c1d0 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f510dffb640 1 -- 172.21.15.161:0/424866407 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110032b40 con 0x7f511815c1d0 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.509+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5110009400 con 0x7f511815c1d0 2024-03-23T14:36:51.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.510+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f511806c6c0 con 0x7f511815c1d0 2024-03-23T14:36:51.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.510+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f511809fe90 con 0x7f511815c1d0 2024-03-23T14:36:51.508 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.511+0000 7f510dffb640 1 -- 172.21.15.161:0/424866407 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5110032ca0 con 0x7f511815c1d0 2024-03-23T14:36:51.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.511+0000 7f510dffb640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50fc06b840 0x7f50fc06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.511+0000 7f510dffb640 1 -- 172.21.15.161:0/424866407 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4943+0+0 (secure 0 0 0) 0x7f51100ab5f0 con 0x7f511815c1d0 2024-03-23T14:36:51.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.512+0000 7f511d200640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50fc06b840 0x7f50fc06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.509 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.512+0000 7f511d200640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50fc06b840 0x7f50fc06dd00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5114001580 tx=0x7f5114000f40 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:51.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.516+0000 7f510dffb640 1 -- 172.21.15.161:0/424866407 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5110076570 con 0x7f511815c1d0 2024-03-23T14:36:51.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.693+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f511804f550 con 0x7f511815c1d0 2024-03-23T14:36:51.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.696+0000 7f510dffb640 1 -- 172.21.15.161:0/424866407 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) ==== 104+0+0 (secure 0 0 0) 0x7f5110079c30 con 0x7f511815c1d0 2024-03-23T14:36:51.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50fc06b840 msgr2=0x7f50fc06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50fc06b840 0x7f50fc06dd00 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f5114001580 tx=0x7f5114000f40 comp rx=0 tx=0).stop 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f511815c1d0 msgr2=0x7f511815e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f511815c1d0 0x7f511815e780 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f5110034040 tx=0x7f51100046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 shutdown_connections 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f50fc06b840 0x7f50fc06dd00 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f511815c1d0 0x7f511815e780 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f51180d0a20 0x7f5118153240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 --2- 172.21.15.161:0/424866407 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f51180ce3c0 0x7f51181f3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 >> 172.21.15.161:0/424866407 conn(0x7f511809aa40 msgr2=0x7f51180cc860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 shutdown_connections 2024-03-23T14:36:51.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.698+0000 7f511f464640 1 -- 172.21.15.161:0/424866407 wait complete. 2024-03-23T14:36:51.705 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification' 2024-03-23T14:36:51.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.810+0000 7fd10af02640 1 Processor -- start 2024-03-23T14:36:51.806 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.810+0000 7fd10af02640 1 -- start start 2024-03-23T14:36:51.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10af02640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd1040724e0 0x7fd10406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10af02640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd10406ead0 0x7fd1040b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.807 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10af02640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd10415be70 con 0x7fd104072100 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10af02640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd10406ef80 con 0x7fd10406ead0 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10af02640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fd10415c430 con 0x7fd1040724e0 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10949f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd10406ead0 0x7fd1040b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd108c9e640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd104072100 0x7fd10415b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48450/0 (socket says 172.21.15.161:48450) 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd108c9e640 1 -- 172.21.15.161:0/4068343335 learned_addr learned my addr 172.21.15.161:0/4068343335 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd103fff640 1 --2- 172.21.15.161:0/4068343335 >> v2:172.21.15.183:3300/0 conn(0x7fd1040724e0 0x7fd10406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10949f640 1 -- 172.21.15.161:0/4068343335 >> v2:172.21.15.183:3300/0 conn(0x7fd1040724e0 msgr2=0x7fd10406e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10949f640 1 --2- 172.21.15.161:0/4068343335 >> v2:172.21.15.183:3300/0 conn(0x7fd1040724e0 0x7fd10406e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10949f640 1 -- 172.21.15.161:0/4068343335 >> v1:172.21.15.161:6789/0 conn(0x7fd104072100 legacy=0x7fd10415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.808 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.811+0000 7fd10949f640 1 -- 172.21.15.161:0/4068343335 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd10415c150 con 0x7fd10406ead0 2024-03-23T14:36:51.809 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.812+0000 7fd10949f640 1 --2- 172.21.15.161:0/4068343335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd10406ead0 0x7fd1040b3470 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd104073420 tx=0x7fd0ec02b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec2fec244bb472cf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.812+0000 7fd1037fe640 1 -- 172.21.15.161:0/4068343335 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec030710 con 0x7fd10406ead0 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.812+0000 7fd1037fe640 1 -- 172.21.15.161:0/4068343335 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0ec02e050 con 0x7fd10406ead0 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.812+0000 7fd1037fe640 1 -- 172.21.15.161:0/4068343335 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0ec030a90 con 0x7fd10406ead0 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.813+0000 7fd10af02640 1 -- 172.21.15.161:0/4068343335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd10406ead0 msgr2=0x7fd1040b3470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.813+0000 7fd10af02640 1 --2- 172.21.15.161:0/4068343335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd10406ead0 0x7fd1040b3470 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fd104073420 tx=0x7fd0ec02b710 comp rx=0 tx=0).stop 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.813+0000 7fd10af02640 1 -- 172.21.15.161:0/4068343335 shutdown_connections 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.813+0000 7fd10af02640 1 --2- 172.21.15.161:0/4068343335 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd10406ead0 0x7fd1040b3470 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.813+0000 7fd10af02640 1 --2- 172.21.15.161:0/4068343335 >> v2:172.21.15.183:3300/0 conn(0x7fd1040724e0 0x7fd10406e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.813+0000 7fd10af02640 1 -- 172.21.15.161:0/4068343335 >> 172.21.15.161:0/4068343335 conn(0x7fd1040b17f0 msgr2=0x7fd1040b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.813+0000 7fd10af02640 1 -- 172.21.15.161:0/4068343335 shutdown_connections 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.813+0000 7fd10af02640 1 -- 172.21.15.161:0/4068343335 wait complete. 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.814+0000 7fd10af02640 1 Processor -- start 2024-03-23T14:36:51.810 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.814+0000 7fd10af02640 1 -- start start 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd10af02640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd10406ead0 0x7fd1041504d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd10af02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd104072100 0x7fd104142120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd10af02640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1040724e0 0x7fd104142660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd10af02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd10414fad0 con 0x7fd104072100 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd10af02640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd104155ab0 con 0x7fd10406ead0 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd10af02640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd104155d90 con 0x7fd1040724e0 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd104072100 0x7fd104142120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd104072100 0x7fd104142120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44910/0 (socket says 172.21.15.161:44910) 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 -- 172.21.15.161:0/2905557930 learned_addr learned my addr 172.21.15.161:0/2905557930 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 -- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1040724e0 msgr2=0x7fd104142660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1040724e0 0x7fd104142660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 -- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd10406ead0 msgr2=0x7fd1041504d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:51.813 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd108c9e640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd10406ead0 0x7fd1041504d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd10406ead0 0x7fd1041504d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 -- 172.21.15.161:0/2905557930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ec009400 con 0x7fd104072100 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.815+0000 7fd103fff640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd104072100 0x7fd104142120 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400d5c0 tx=0x7fd0f400da80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.816+0000 7fd1017fa640 1 -- 172.21.15.161:0/2905557930 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f40125f0 con 0x7fd104072100 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.816+0000 7fd1017fa640 1 -- 172.21.15.161:0/2905557930 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0f4012750 con 0x7fd104072100 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.816+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd104156070 con 0x7fd104072100 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.816+0000 7fd1017fa640 1 -- 172.21.15.161:0/2905557930 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0f4012b00 con 0x7fd104072100 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.816+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd104150a10 con 0x7fd104072100 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.817+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1040735e0 con 0x7fd104072100 2024-03-23T14:36:51.814 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.817+0000 7fd1017fa640 1 -- 172.21.15.161:0/2905557930 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd0f4012c60 con 0x7fd104072100 2024-03-23T14:36:51.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.817+0000 7fd1017fa640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd0e406b810 0x7fd0e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:51.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.818+0000 7fd1017fa640 1 -- 172.21.15.161:0/2905557930 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4943+0+0 (secure 0 0 0) 0x7fd0f408a3a0 con 0x7fd104072100 2024-03-23T14:36:51.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.818+0000 7fd108c9e640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd0e406b810 0x7fd0e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:51.815 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.818+0000 7fd108c9e640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd0e406b810 0x7fd0e406dcd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0012e0 tx=0x7fd0fc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:51.817 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:51.822+0000 7fd1017fa640 1 -- 172.21.15.161:0/2905557930 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd0f40550c0 con 0x7fd104072100 2024-03-23T14:36:52.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.065+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]} v 0) -- 0x7fd1040b4270 con 0x7fd104072100 2024-03-23T14:36:52.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.124+0000 7fd1017fa640 1 -- 172.21.15.161:0/2905557930 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_forbidden_modification"]}]=0 v114) ==== 147+0+0 (secure 0 0 0) 0x7fd0f4058780 con 0x7fd104072100 2024-03-23T14:36:52.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.127+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd0e406b810 msgr2=0x7fd0e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.127+0000 7fd10af02640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd0e406b810 0x7fd0e406dcd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd0fc0012e0 tx=0x7fd0fc009040 comp rx=0 tx=0).stop 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd104072100 msgr2=0x7fd104142120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd104072100 0x7fd104142120 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fd0f400d5c0 tx=0x7fd0f400da80 comp rx=0 tx=0).stop 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 shutdown_connections 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd0e406b810 0x7fd0e406dcd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd1040724e0 0x7fd104142660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd104072100 0x7fd104142120 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 --2- 172.21.15.161:0/2905557930 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd10406ead0 0x7fd1041504d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 >> 172.21.15.161:0/2905557930 conn(0x7fd1040b17f0 msgr2=0x7fd10415c570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 shutdown_connections 2024-03-23T14:36:52.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.128+0000 7fd10af02640 1 -- 172.21.15.161:0/2905557930 wait complete. 2024-03-23T14:36:52.134 INFO:tasks.cephfs_test_runner:That a client who does not have the capability for setting ... ok 2024-03-23T14:36:52.135 INFO:tasks.cephfs_test_runner:Starting test: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-23T14:36:52.135 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:52.235 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.240+0000 7fad5d834640 1 Processor -- start 2024-03-23T14:36:52.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.240+0000 7fad5d834640 1 -- start start 2024-03-23T14:36:52.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.240+0000 7fad5d834640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5815e400 0x7fad580cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.240+0000 7fad5d834640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fad580d2b40 0x7fad580b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.240+0000 7fad5d834640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad580d0460 con 0x7fad580cf9d0 2024-03-23T14:36:52.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.240+0000 7fad5d834640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fad580d1050 con 0x7fad580d2b40 2024-03-23T14:36:52.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.240+0000 7fad5d834640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fad580d06b0 con 0x7fad5815e400 2024-03-23T14:36:52.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad56ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5815e400 0x7fad580cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad577fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fad580d2b40 0x7fad580b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.237 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad567fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fad580cf9d0 0x7fad580d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:54758/0 (socket says 172.21.15.161:54758) 2024-03-23T14:36:52.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad567fc640 1 -- 172.21.15.161:0/3654148847 learned_addr learned my addr 172.21.15.161:0/3654148847 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:52.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad577fe640 1 -- 172.21.15.161:0/3654148847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5815e400 msgr2=0x7fad580cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:52.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad577fe640 1 --2- 172.21.15.161:0/3654148847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5815e400 0x7fad580cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad577fe640 1 -- 172.21.15.161:0/3654148847 >> v1:172.21.15.183:6789/0 conn(0x7fad580cf9d0 legacy=0x7fad580d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad577fe640 1 -- 172.21.15.161:0/3654148847 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad580d0990 con 0x7fad580d2b40 2024-03-23T14:36:52.238 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.241+0000 7fad56ffd640 1 --2- 172.21.15.161:0/3654148847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5815e400 0x7fad580cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:52.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.242+0000 7fad577fe640 1 --2- 172.21.15.161:0/3654148847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580d2b40 0x7fad580b0a80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fad50009d90 tx=0x7fad5002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12fefe85f9f53ee server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.239 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.242+0000 7fad55ffb640 1 -- 172.21.15.161:0/3654148847 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad5002cc50 con 0x7fad580d2b40 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.242+0000 7fad55ffb640 1 -- 172.21.15.161:0/3654148847 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad5002cdb0 con 0x7fad580d2b40 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.242+0000 7fad55ffb640 1 -- 172.21.15.161:0/3654148847 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50034660 con 0x7fad580d2b40 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.243+0000 7fad5d834640 1 -- 172.21.15.161:0/3654148847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580d2b40 msgr2=0x7fad580b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.243+0000 7fad5d834640 1 --2- 172.21.15.161:0/3654148847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580d2b40 0x7fad580b0a80 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fad50009d90 tx=0x7fad5002bea0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.243+0000 7fad5d834640 1 -- 172.21.15.161:0/3654148847 shutdown_connections 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.243+0000 7fad5d834640 1 --2- 172.21.15.161:0/3654148847 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580d2b40 0x7fad580b0a80 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.243+0000 7fad5d834640 1 --2- 172.21.15.161:0/3654148847 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad5815e400 0x7fad580cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.243+0000 7fad5d834640 1 -- 172.21.15.161:0/3654148847 >> 172.21.15.161:0/3654148847 conn(0x7fad5809d190 msgr2=0x7fad580d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.243+0000 7fad5d834640 1 -- 172.21.15.161:0/3654148847 shutdown_connections 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.243+0000 7fad5d834640 1 -- 172.21.15.161:0/3654148847 wait complete. 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.244+0000 7fad5d834640 1 Processor -- start 2024-03-23T14:36:52.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.244+0000 7fad5d834640 1 -- start start 2024-03-23T14:36:52.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad5d834640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580cf9d0 0x7fad5815d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad5d834640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad580d2b40 0x7fad581f4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad5d834640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad5815e400 0x7fad580ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad5d834640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fad580b2d30 con 0x7fad5815e400 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad5d834640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fad580b3010 con 0x7fad580cf9d0 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad5d834640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fad580b32f0 con 0x7fad580d2b40 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad5815e400 0x7fad580ccd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad5815e400 0x7fad580ccd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44916/0 (socket says 172.21.15.161:44916) 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 -- 172.21.15.161:0/1281633109 learned_addr learned my addr 172.21.15.161:0/1281633109 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad56ffd640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580cf9d0 0x7fad5815d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad567fc640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad580d2b40 0x7fad581f4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 -- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad580d2b40 msgr2=0x7fad581f4e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad580d2b40 0x7fad581f4e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 -- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580cf9d0 msgr2=0x7fad5815d3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580cf9d0 0x7fad5815d3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 -- 172.21.15.161:0/1281633109 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad48009400 con 0x7fad5815e400 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.245+0000 7fad577fe640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad5815e400 0x7fad580ccd30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fad50009d90 tx=0x7fad50004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.246+0000 7fad3ffff640 1 -- 172.21.15.161:0/1281633109 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50040650 con 0x7fad5815e400 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.246+0000 7fad3ffff640 1 -- 172.21.15.161:0/1281633109 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad500407d0 con 0x7fad5815e400 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.246+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad50009a70 con 0x7fad5815e400 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.246+0000 7fad3ffff640 1 -- 172.21.15.161:0/1281633109 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad50040b80 con 0x7fad5815e400 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.246+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad580b38b0 con 0x7fad5815e400 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.247+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad580b3aa0 con 0x7fad5815e400 2024-03-23T14:36:52.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.247+0000 7fad3ffff640 1 -- 172.21.15.161:0/1281633109 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fad50040ce0 con 0x7fad5815e400 2024-03-23T14:36:52.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.247+0000 7fad3ffff640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fad4406b810 0x7fad4406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.248+0000 7fad3ffff640 1 -- 172.21.15.161:0/1281633109 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4943+0+0 (secure 0 0 0) 0x7fad500acbb0 con 0x7fad5815e400 2024-03-23T14:36:52.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.248+0000 7fad56ffd640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fad4406b810 0x7fad4406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.248+0000 7fad56ffd640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fad4406b810 0x7fad4406dcd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fad48009c30 tx=0x7fad48009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.252+0000 7fad3ffff640 1 -- 172.21.15.161:0/1281633109 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fad50077990 con 0x7fad5815e400 2024-03-23T14:36:52.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.426+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fad580a1e00 con 0x7fad5815e400 2024-03-23T14:36:52.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.426+0000 7fad3ffff640 1 -- 172.21.15.161:0/1281633109 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v40) ==== 74+0+13058 (secure 0 0 0) 0x7fad5007b050 con 0x7fad5815e400 2024-03-23T14:36:52.422 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:52.423 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":40,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:51.119403+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.183:6836/2372368263":"2024-03-24T14:36:47.098104+0000","172.21.15.183:6834/2372368263":"2024-03-24T14:36:47.098104+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fad4406b810 msgr2=0x7fad4406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fad4406b810 0x7fad4406dcd0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fad48009c30 tx=0x7fad48009190 comp rx=0 tx=0).stop 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad5815e400 msgr2=0x7fad580ccd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad5815e400 0x7fad580ccd30 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fad50009d90 tx=0x7fad50004a40 comp rx=0 tx=0).stop 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 shutdown_connections 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fad4406b810 0x7fad4406dcd0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fad5815e400 0x7fad580ccd30 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fad580d2b40 0x7fad581f4e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 --2- 172.21.15.161:0/1281633109 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fad580cf9d0 0x7fad5815d3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 >> 172.21.15.161:0/1281633109 conn(0x7fad5809d190 msgr2=0x7fad580b8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 shutdown_connections 2024-03-23T14:36:52.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.429+0000 7fad5d834640 1 -- 172.21.15.161:0/1281633109 wait complete. 2024-03-23T14:36:52.436 INFO:tasks.ceph_test_case.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:52.436 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:52.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.543+0000 7f2f4fc23640 1 Processor -- start 2024-03-23T14:36:52.539 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.544+0000 7f2f4fc23640 1 -- start start 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.544+0000 7f2f4fc23640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f480d1270 0x7f2f480d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.544+0000 7f2f4fc23640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2f480d29d0 0x7f2f480b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.544+0000 7f2f4fc23640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f48161c00 con 0x7f2f48161820 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.544+0000 7f2f4fc23640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2f480d2e20 con 0x7f2f480d29d0 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.544+0000 7f2f4fc23640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2f480cea30 con 0x7f2f480d1270 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.545+0000 7f2f4e1c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2f480d29d0 0x7f2f480b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.545+0000 7f2f4d9bf640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f2f48161820 0x7f2f480d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48460/0 (socket says 172.21.15.161:48460) 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.545+0000 7f2f4d9bf640 1 -- 172.21.15.161:0/1938746882 learned_addr learned my addr 172.21.15.161:0/1938746882 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.545+0000 7f2f4e1c0640 1 -- 172.21.15.161:0/1938746882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f480d1270 msgr2=0x7f2f480d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.545+0000 7f2f4e1c0640 1 --2- 172.21.15.161:0/1938746882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f480d1270 0x7f2f480d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.545+0000 7f2f4e1c0640 1 -- 172.21.15.161:0/1938746882 >> v1:172.21.15.161:6789/0 conn(0x7f2f48161820 legacy=0x7f2f480d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.545+0000 7f2f4c9bd640 1 -- 172.21.15.161:0/1938746882 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 466750947 0 0) 0x7f2f48161c00 con 0x7f2f48161820 2024-03-23T14:36:52.541 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.545+0000 7f2f4e1c0640 1 -- 172.21.15.161:0/1938746882 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f480cfd70 con 0x7f2f480d29d0 2024-03-23T14:36:52.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4e1c0640 1 --2- 172.21.15.161:0/1938746882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f480d29d0 0x7f2f480b0a80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009d90 tx=0x7f2f4002bea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e47a4c42a06f17c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4c9bd640 1 -- 172.21.15.161:0/1938746882 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4002cc50 con 0x7f2f480d29d0 2024-03-23T14:36:52.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4c9bd640 1 -- 172.21.15.161:0/1938746882 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f4002cdb0 con 0x7f2f480d29d0 2024-03-23T14:36:52.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4c9bd640 1 -- 172.21.15.161:0/1938746882 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f40034660 con 0x7f2f480d29d0 2024-03-23T14:36:52.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1938746882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f480d29d0 msgr2=0x7f2f480b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1938746882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f480d29d0 0x7f2f480b0a80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f2f40009d90 tx=0x7f2f4002bea0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1938746882 shutdown_connections 2024-03-23T14:36:52.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1938746882 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f480d29d0 0x7f2f480b0a80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1938746882 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f480d1270 0x7f2f480d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1938746882 >> 172.21.15.161:0/1938746882 conn(0x7f2f4809d190 msgr2=0x7f2f480d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:52.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1938746882 shutdown_connections 2024-03-23T14:36:52.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.546+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1938746882 wait complete. 2024-03-23T14:36:52.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.547+0000 7f2f4fc23640 1 Processor -- start 2024-03-23T14:36:52.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4fc23640 1 -- start start 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4fc23640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f480d1270 0x7f2f480c9650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4fc23640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f480d29d0 0x7f2f48149e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4fc23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f48161820 0x7f2f4814a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4fc23640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2f4815c310 con 0x7f2f48161820 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4fc23640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2f480ca840 con 0x7f2f480d29d0 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4fc23640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2f480c9fa0 con 0x7f2f480d1270 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4e1c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f48161820 0x7f2f4814a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4e1c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f48161820 0x7f2f4814a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44926/0 (socket says 172.21.15.161:44926) 2024-03-23T14:36:52.546 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.548+0000 7f2f4e1c0640 1 -- 172.21.15.161:0/1645606227 learned_addr learned my addr 172.21.15.161:0/1645606227 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f4e1c0640 1 -- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f480d1270 msgr2=0x7f2f480c9650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f4e1c0640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f480d1270 0x7f2f480c9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f4e1c0640 1 -- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f480d29d0 msgr2=0x7f2f48149e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f4e1c0640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f480d29d0 0x7f2f48149e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f4e1c0640 1 -- 172.21.15.161:0/1645606227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f40009a70 con 0x7f2f48161820 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f4e1c0640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f48161820 0x7f2f4814a3c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2f40037040 tx=0x7f2f40004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f3e7fc640 1 -- 172.21.15.161:0/1645606227 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4003e610 con 0x7f2f48161820 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f3e7fc640 1 -- 172.21.15.161:0/1645606227 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f4003e790 con 0x7f2f48161820 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f480ca280 con 0x7f2f48161820 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f3e7fc640 1 -- 172.21.15.161:0/1645606227 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f4003eb40 con 0x7f2f48161820 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.549+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f480c7650 con 0x7f2f48161820 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.550+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f4804f4e0 con 0x7f2f48161820 2024-03-23T14:36:52.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.550+0000 7f2f3e7fc640 1 -- 172.21.15.161:0/1645606227 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2f4003b050 con 0x7f2f48161820 2024-03-23T14:36:52.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.551+0000 7f2f3e7fc640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2c06b810 0x7f2f2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.551+0000 7f2f3e7fc640 1 -- 172.21.15.161:0/1645606227 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4943+0+0 (secure 0 0 0) 0x7f2f400ac190 con 0x7f2f48161820 2024-03-23T14:36:52.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.551+0000 7f2f4d9bf640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2c06b810 0x7f2f2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.551+0000 7f2f4d9bf640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2c06b810 0x7f2f2c06dcd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2f44007fd0 tx=0x7f2f44007920 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.555+0000 7f2f3e7fc640 1 -- 172.21.15.161:0/1645606227 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2f40077060 con 0x7f2f48161820 2024-03-23T14:36:52.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.723+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f2f4815c450 con 0x7f2f48161820 2024-03-23T14:36:52.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.723+0000 7f2f3e7fc640 1 -- 172.21.15.161:0/1645606227 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v40) ==== 93+0+10 (secure 0 0 0) 0x7f2f4007a720 con 0x7f2f48161820 2024-03-23T14:36:52.719 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:52.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2c06b810 msgr2=0x7f2f2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2c06b810 0x7f2f2c06dcd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f2f44007fd0 tx=0x7f2f44007920 comp rx=0 tx=0).stop 2024-03-23T14:36:52.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f48161820 msgr2=0x7f2f4814a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f48161820 0x7f2f4814a3c0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f2f40037040 tx=0x7f2f40004a40 comp rx=0 tx=0).stop 2024-03-23T14:36:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 shutdown_connections 2024-03-23T14:36:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2f2c06b810 0x7f2f2c06dcd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2f48161820 0x7f2f4814a3c0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2f480d29d0 0x7f2f48149e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 --2- 172.21.15.161:0/1645606227 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2f480d1270 0x7f2f480c9650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 >> 172.21.15.161:0/1645606227 conn(0x7f2f4809d190 msgr2=0x7f2f4806c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 shutdown_connections 2024-03-23T14:36:52.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.725+0000 7f2f4fc23640 1 -- 172.21.15.161:0/1645606227 wait complete. 2024-03-23T14:36:52.732 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-23T14:36:52.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.837+0000 7f54f61a7640 1 Processor -- start 2024-03-23T14:36:52.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.837+0000 7f54f61a7640 1 -- start start 2024-03-23T14:36:52.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54f61a7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f01552f0 0x7f54f00b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54f61a7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f54f00b2aa0 0x7f54f00cfc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.835 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54f61a7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f54f00b4250 0x7f54f00d01b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54f61a7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f54f015f480 con 0x7f54f00b2aa0 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54f61a7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f54f01556d0 con 0x7f54f00b4250 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54f61a7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f54f015e870 con 0x7f54f01552f0 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54eeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f54f00b2aa0 0x7f54f00cfc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54eeffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f54f00b2aa0 0x7f54f00cfc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44938/0 (socket says 172.21.15.161:44938) 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54eeffd640 1 -- 172.21.15.161:0/890670513 learned_addr learned my addr 172.21.15.161:0/890670513 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54effff640 1 --2- 172.21.15.161:0/890670513 >> v2:172.21.15.183:3300/0 conn(0x7f54f00b4250 0x7f54f00d01b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54ef7fe640 1 --2- 172.21.15.161:0/890670513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f01552f0 0x7f54f00b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54eeffd640 1 -- 172.21.15.161:0/890670513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f01552f0 msgr2=0x7f54f00b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54eeffd640 1 --2- 172.21.15.161:0/890670513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f01552f0 0x7f54f00b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54eeffd640 1 -- 172.21.15.161:0/890670513 >> v2:172.21.15.183:3300/0 conn(0x7f54f00b4250 msgr2=0x7f54f00d01b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54eeffd640 1 --2- 172.21.15.161:0/890670513 >> v2:172.21.15.183:3300/0 conn(0x7f54f00b4250 0x7f54f00d01b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.836 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.838+0000 7f54eeffd640 1 -- 172.21.15.161:0/890670513 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54f015eb50 con 0x7f54f00b2aa0 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54eeffd640 1 --2- 172.21.15.161:0/890670513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f00cfc70 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f54f00b1630 tx=0x7f54e002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd696bfdd2bf949d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54ee7fc640 1 -- 172.21.15.161:0/890670513 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e0031870 con 0x7f54f00b2aa0 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54ee7fc640 1 -- 172.21.15.161:0/890670513 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54e00319d0 con 0x7f54f00b2aa0 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54ee7fc640 1 -- 172.21.15.161:0/890670513 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e0031d60 con 0x7f54f00b2aa0 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54f61a7640 1 -- 172.21.15.161:0/890670513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 msgr2=0x7f54f00cfc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54f61a7640 1 --2- 172.21.15.161:0/890670513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f00cfc70 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f54f00b1630 tx=0x7f54e002b910 comp rx=0 tx=0).stop 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54f61a7640 1 -- 172.21.15.161:0/890670513 shutdown_connections 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54f61a7640 1 --2- 172.21.15.161:0/890670513 >> v2:172.21.15.183:3300/0 conn(0x7f54f00b4250 0x7f54f00d01b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54f61a7640 1 --2- 172.21.15.161:0/890670513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f00cfc70 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54f61a7640 1 --2- 172.21.15.161:0/890670513 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f01552f0 0x7f54f00b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.839+0000 7f54f61a7640 1 -- 172.21.15.161:0/890670513 >> 172.21.15.161:0/890670513 conn(0x7f54f006b260 msgr2=0x7f54f006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.840+0000 7f54f61a7640 1 -- 172.21.15.161:0/890670513 shutdown_connections 2024-03-23T14:36:52.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.840+0000 7f54f61a7640 1 -- 172.21.15.161:0/890670513 wait complete. 2024-03-23T14:36:52.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.841+0000 7f54f61a7640 1 Processor -- start 2024-03-23T14:36:52.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.841+0000 7f54f61a7640 1 -- start start 2024-03-23T14:36:52.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.841+0000 7f54f61a7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f014c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.841+0000 7f54f61a7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f00b4250 0x7f54f01f18d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.841+0000 7f54f61a7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54f01552f0 0x7f54f01f23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54f61a7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f54f01f2900 con 0x7f54f00b2aa0 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54f61a7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f54f0143dd0 con 0x7f54f01552f0 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54f61a7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f54f01559b0 con 0x7f54f00b4250 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f014c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f014c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44942/0 (socket says 172.21.15.161:44942) 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 -- 172.21.15.161:0/2934549342 learned_addr learned my addr 172.21.15.161:0/2934549342 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54eeffd640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f00b4250 0x7f54f01f18d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54effff640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54f01552f0 0x7f54f01f23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 -- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f00b4250 msgr2=0x7f54f01f18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f00b4250 0x7f54f01f18d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 -- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54f01552f0 msgr2=0x7f54f01f23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54f01552f0 0x7f54f01f23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 -- 172.21.15.161:0/2934549342 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54e00094d0 con 0x7f54f00b2aa0 2024-03-23T14:36:52.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54eeffd640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f00b4250 0x7f54f01f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:52.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54effff640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54f01552f0 0x7f54f01f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:52.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54ef7fe640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f014c540 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f54e400d5f0 tx=0x7f54e400dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54d7fff640 1 -- 172.21.15.161:0/2934549342 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e40125f0 con 0x7f54f00b2aa0 2024-03-23T14:36:52.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54d7fff640 1 -- 172.21.15.161:0/2934549342 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54e4012750 con 0x7f54f00b2aa0 2024-03-23T14:36:52.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54f014d6a0 con 0x7f54f00b2aa0 2024-03-23T14:36:52.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54d7fff640 1 -- 172.21.15.161:0/2934549342 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54e4012b00 con 0x7f54f00b2aa0 2024-03-23T14:36:52.842 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.842+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54f014d0e0 con 0x7f54f00b2aa0 2024-03-23T14:36:52.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.843+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54f0159290 con 0x7f54f00b2aa0 2024-03-23T14:36:52.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.844+0000 7f54d7fff640 1 -- 172.21.15.161:0/2934549342 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f54e4012c60 con 0x7f54f00b2aa0 2024-03-23T14:36:52.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.844+0000 7f54d7fff640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f54dc06b810 0x7f54dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:52.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.844+0000 7f54d7fff640 1 -- 172.21.15.161:0/2934549342 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4943+0+0 (secure 0 0 0) 0x7f54e408a370 con 0x7f54f00b2aa0 2024-03-23T14:36:52.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.844+0000 7f54eeffd640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f54dc06b810 0x7f54dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:52.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.845+0000 7f54eeffd640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f54dc06b810 0x7f54dc06dcd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f54e0036000 tx=0x7f54e003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:52.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:52.848+0000 7f54d7fff640 1 -- 172.21.15.161:0/2934549342 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f54e4055090 con 0x7f54f00b2aa0 2024-03-23T14:36:53.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.085+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7f54f004f4e0 con 0x7f54f00b2aa0 2024-03-23T14:36:53.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.129+0000 7f54d7fff640 1 -- 172.21.15.161:0/2934549342 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v115) ==== 137+0+0 (secure 0 0 0) 0x7f54e405de60 con 0x7f54f00b2aa0 2024-03-23T14:36:53.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f54dc06b810 msgr2=0x7f54dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f54dc06b810 0x7f54dc06dcd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f54e0036000 tx=0x7f54e003c040 comp rx=0 tx=0).stop 2024-03-23T14:36:53.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 msgr2=0x7f54f014c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f014c540 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f54e400d5f0 tx=0x7f54e400dab0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 shutdown_connections 2024-03-23T14:36:53.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f54dc06b810 0x7f54dc06dcd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f54f01552f0 0x7f54f01f23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f54f00b4250 0x7f54f01f18d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 --2- 172.21.15.161:0/2934549342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f54f00b2aa0 0x7f54f014c540 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 >> 172.21.15.161:0/2934549342 conn(0x7f54f006b260 msgr2=0x7f54f00ceff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:53.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 shutdown_connections 2024-03-23T14:36:53.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.131+0000 7f54f61a7640 1 -- 172.21.15.161:0/2934549342 wait complete. 2024-03-23T14:36:53.138 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2024-03-23T14:36:53.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.246+0000 7f822cac9640 1 Processor -- start 2024-03-23T14:36:53.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.246+0000 7f822cac9640 1 -- start start 2024-03-23T14:36:53.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.247+0000 7f822cac9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f82280d1cb0 0x7f82280cf220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.247+0000 7f822cac9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82280d1330 con 0x7f8228161750 2024-03-23T14:36:53.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.247+0000 7f822cac9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82280d0720 con 0x7f82280d2cd0 2024-03-23T14:36:53.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.247+0000 7f822cac9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f82280d0a00 con 0x7f82280d1cb0 2024-03-23T14:36:53.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.247+0000 7f822659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8228161750 0x7f82280d25c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48468/0 (socket says 172.21.15.161:48468) 2024-03-23T14:36:53.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.247+0000 7f822659c640 1 -- 172.21.15.161:0/2478513532 learned_addr learned my addr 172.21.15.161:0/2478513532 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:53.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.247+0000 7f8226d9d640 1 --2- 172.21.15.161:0/2478513532 >> v2:172.21.15.161:3300/0 conn(0x7f82280d1cb0 0x7f82280cf220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.248+0000 7f8226d9d640 1 -- 172.21.15.161:0/2478513532 >> v1:172.21.15.183:6789/0 conn(0x7f82280d2cd0 legacy=0x7f82280ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:53.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.248+0000 7f8226d9d640 1 -- 172.21.15.161:0/2478513532 >> v1:172.21.15.161:6789/0 conn(0x7f8228161750 legacy=0x7f82280d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.248+0000 7f8226d9d640 1 -- 172.21.15.161:0/2478513532 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82280d0ce0 con 0x7f82280d1cb0 2024-03-23T14:36:53.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.248+0000 7f8226d9d640 1 --2- 172.21.15.161:0/2478513532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82280d1cb0 0x7f82280cf220 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f82200097a0 tx=0x7f822002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d13d2e2bca5343d6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:53.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.248+0000 7f822559a640 1 -- 172.21.15.161:0/2478513532 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220031830 con 0x7f82280d1cb0 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822559a640 1 -- 172.21.15.161:0/2478513532 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8220031990 con 0x7f82280d1cb0 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822559a640 1 -- 172.21.15.161:0/2478513532 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8220031d20 con 0x7f82280d1cb0 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822cac9640 1 -- 172.21.15.161:0/2478513532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82280d1cb0 msgr2=0x7f82280cf220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822cac9640 1 --2- 172.21.15.161:0/2478513532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82280d1cb0 0x7f82280cf220 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f82200097a0 tx=0x7f822002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822cac9640 1 -- 172.21.15.161:0/2478513532 shutdown_connections 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822cac9640 1 --2- 172.21.15.161:0/2478513532 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82280d1cb0 0x7f82280cf220 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822cac9640 1 -- 172.21.15.161:0/2478513532 >> 172.21.15.161:0/2478513532 conn(0x7f822809d450 msgr2=0x7f82280d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822cac9640 1 -- 172.21.15.161:0/2478513532 shutdown_connections 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.249+0000 7f822cac9640 1 -- 172.21.15.161:0/2478513532 wait complete. 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.250+0000 7f822cac9640 1 Processor -- start 2024-03-23T14:36:53.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.250+0000 7f822cac9640 1 -- start start 2024-03-23T14:36:53.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f822cac9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82280d1cb0 0x7f82280cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f822cac9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280d2cd0 0x7f82280cc650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f822cac9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8228161750 0x7f82280ccb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f822cac9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f82280b89e0 con 0x7f8228161750 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f822cac9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f82280cbc80 con 0x7f82280d2cd0 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f822cac9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f82280cd0d0 con 0x7f82280d1cb0 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f8226d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8228161750 0x7f82280ccb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f8226d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8228161750 0x7f82280ccb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44966/0 (socket says 172.21.15.161:44966) 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.251+0000 7f8226d9d640 1 -- 172.21.15.161:0/1288385514 learned_addr learned my addr 172.21.15.161:0/1288385514 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f822659c640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82280d1cb0 0x7f82280cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8225d9b640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280d2cd0 0x7f82280cc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8226d9d640 1 -- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82280d1cb0 msgr2=0x7f82280cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8226d9d640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82280d1cb0 0x7f82280cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8226d9d640 1 -- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280d2cd0 msgr2=0x7f82280cc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8226d9d640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280d2cd0 0x7f82280cc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8226d9d640 1 -- 172.21.15.161:0/1288385514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8220009400 con 0x7f8228161750 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f822659c640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82280d1cb0 0x7f82280cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8225d9b640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280d2cd0 0x7f82280cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8226d9d640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8228161750 0x7f82280ccb90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f822002bee0 tx=0x7f82200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8216ffd640 1 -- 172.21.15.161:0/1288385514 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f822003b9f0 con 0x7f8228161750 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8216ffd640 1 -- 172.21.15.161:0/1288385514 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f822003bb50 con 0x7f8228161750 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f8216ffd640 1 -- 172.21.15.161:0/1288385514 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82200433f0 con 0x7f8228161750 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82280cb350 con 0x7f8228161750 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.252+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82280cb910 con 0x7f8228161750 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.253+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82280cbe30 con 0x7f8228161750 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.253+0000 7f8216ffd640 1 -- 172.21.15.161:0/1288385514 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f822004c3b0 con 0x7f8228161750 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.254+0000 7f8216ffd640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f820806b810 0x7f820806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.254+0000 7f8216ffd640 1 -- 172.21.15.161:0/1288385514 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4943+0+0 (secure 0 0 0) 0x7f82200abdf0 con 0x7f8228161750 2024-03-23T14:36:53.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.254+0000 7f822659c640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f820806b810 0x7f820806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.255+0000 7f822659c640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f820806b810 0x7f820806dcd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f82180012e0 tx=0x7f8218009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:53.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.258+0000 7f8216ffd640 1 -- 172.21.15.161:0/1288385514 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8220076cc0 con 0x7f8228161750 2024-03-23T14:36:53.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.430+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) -- 0x7f82280a2030 con 0x7f8228161750 2024-03-23T14:36:53.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.433+0000 7f8216ffd640 1 -- 172.21.15.161:0/1288385514 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) ==== 122+0+0 (secure 0 0 0) 0x7f82200480c0 con 0x7f8228161750 2024-03-23T14:36:53.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f820806b810 msgr2=0x7f820806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f820806b810 0x7f820806dcd0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f82180012e0 tx=0x7f8218009040 comp rx=0 tx=0).stop 2024-03-23T14:36:53.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8228161750 msgr2=0x7f82280ccb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8228161750 0x7f82280ccb90 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f822002bee0 tx=0x7f82200046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 shutdown_connections 2024-03-23T14:36:53.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f820806b810 0x7f820806dcd0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8228161750 0x7f82280ccb90 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82280d2cd0 0x7f82280cc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 --2- 172.21.15.161:0/1288385514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82280d1cb0 0x7f82280cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 >> 172.21.15.161:0/1288385514 conn(0x7f822809d450 msgr2=0x7f82280b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:53.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 shutdown_connections 2024-03-23T14:36:53.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.435+0000 7f822cac9640 1 -- 172.21.15.161:0/1288385514 wait complete. 2024-03-23T14:36:53.442 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T14:36:53.442 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T14:36:53.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T14:36:53.541 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T14:36:53.542 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:53.542 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:53.557 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18767 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T14:36:53.557 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:36:53.557 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:53.575 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056359,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17751 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T14:36:53.575 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:36:53.575 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:53.611 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18767 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T14:36:53.612 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:36:53.612 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:36:53.632 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056359,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17751 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T14:36:53.632 INFO:tasks.cephfs.cephfs_test_case:[, , , ] 2024-03-23T14:36:53.632 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:36:53.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.756+0000 7f00e260e640 1 Processor -- start 2024-03-23T14:36:53.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.756+0000 7f00e260e640 1 -- start start 2024-03-23T14:36:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.756+0000 7f00e260e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f00dc0d2b40 0x7f00dc0b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.756+0000 7f00e260e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00dc0d0460 con 0x7f00dc15e400 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.756+0000 7f00e260e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00dc0d06b0 con 0x7f00dc0d1470 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.756+0000 7f00e260e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f00dc0d1050 con 0x7f00dc0d2b40 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.757+0000 7f00e0bab640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f00dc0d2b40 0x7f00dc0b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.757+0000 7f00e0bab640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f00dc0d2b40 0x7f00dc0b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44978/0 (socket says 172.21.15.161:44978) 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.757+0000 7f00dbfff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f00dc15e400 0x7f00dc0cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48474/0 (socket says 172.21.15.161:48474) 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.757+0000 7f00e0bab640 1 -- 172.21.15.161:0/2001638358 learned_addr learned my addr 172.21.15.161:0/2001638358 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.757+0000 7f00e0bab640 1 -- 172.21.15.161:0/2001638358 >> v1:172.21.15.183:6789/0 conn(0x7f00dc0d1470 legacy=0x7f00dc0d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.757+0000 7f00e0bab640 1 -- 172.21.15.161:0/2001638358 >> v1:172.21.15.161:6789/0 conn(0x7f00dc15e400 legacy=0x7f00dc0cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.757+0000 7f00e0bab640 1 -- 172.21.15.161:0/2001638358 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00dc0d0990 con 0x7f00dc0d2b40 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.757+0000 7f00daffd640 1 -- 172.21.15.161:0/2001638358 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553678924 0 0) 0x7f00dc0d0460 con 0x7f00dc15e400 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00e0bab640 1 --2- 172.21.15.161:0/2001638358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d2b40 0x7f00dc0b0a80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009e50 tx=0x7f00d402bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feb904f34f07e04e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00daffd640 1 -- 172.21.15.161:0/2001638358 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d402cb20 con 0x7f00dc0d2b40 2024-03-23T14:36:53.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00daffd640 1 -- 172.21.15.161:0/2001638358 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d402cc80 con 0x7f00dc0d2b40 2024-03-23T14:36:53.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00daffd640 1 -- 172.21.15.161:0/2001638358 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d40343f0 con 0x7f00dc0d2b40 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00e260e640 1 -- 172.21.15.161:0/2001638358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d2b40 msgr2=0x7f00dc0b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00e260e640 1 --2- 172.21.15.161:0/2001638358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d2b40 0x7f00dc0b0a80 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f00d4009e50 tx=0x7f00d402bf80 comp rx=0 tx=0).stop 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00e260e640 1 -- 172.21.15.161:0/2001638358 shutdown_connections 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00e260e640 1 --2- 172.21.15.161:0/2001638358 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d2b40 0x7f00dc0b0a80 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00e260e640 1 -- 172.21.15.161:0/2001638358 >> 172.21.15.161:0/2001638358 conn(0x7f00dc09d190 msgr2=0x7f00dc0d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00e260e640 1 -- 172.21.15.161:0/2001638358 shutdown_connections 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.758+0000 7f00e260e640 1 -- 172.21.15.161:0/2001638358 wait complete. 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.759+0000 7f00e260e640 1 Processor -- start 2024-03-23T14:36:53.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.760+0000 7f00e260e640 1 -- start start 2024-03-23T14:36:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.760+0000 7f00e260e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d1470 0x7f00dc1445f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.760+0000 7f00e260e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00dc0d2b40 0x7f00dc15c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.760+0000 7f00e260e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00dc15e400 0x7f00dc0c9b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.760+0000 7f00e260e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc144b30 con 0x7f00dc0d1470 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.760+0000 7f00e260e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f00dc143ba0 con 0x7f00dc15e400 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.760+0000 7f00e260e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f00dc0ca730 con 0x7f00dc0d2b40 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d1470 0x7f00dc1445f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d1470 0x7f00dc1445f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:44992/0 (socket says 172.21.15.161:44992) 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 -- 172.21.15.161:0/200332621 learned_addr learned my addr 172.21.15.161:0/200332621 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00e0bab640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00dc15e400 0x7f00dc0c9b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 -- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00dc0d2b40 msgr2=0x7f00dc15c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00dc0d2b40 0x7f00dc15c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 -- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00dc15e400 msgr2=0x7f00dc0c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00dc15e400 0x7f00dc0c9b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 -- 172.21.15.161:0/200332621 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d4009b30 con 0x7f00dc0d1470 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00dbfff640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d1470 0x7f00dc1445f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f00d000c4d0 tx=0x7f00d000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00d8ff9640 1 -- 172.21.15.161:0/200332621 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d00074e0 con 0x7f00dc0d1470 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00d8ff9640 1 -- 172.21.15.161:0/200332621 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00d0007640 con 0x7f00dc0d1470 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00dc0caa10 con 0x7f00dc0d1470 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00d8ff9640 1 -- 172.21.15.161:0/200332621 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00d0004a20 con 0x7f00dc0d1470 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.761+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00dc1440d0 con 0x7f00dc0d1470 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.762+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00dc0a1e00 con 0x7f00dc0d1470 2024-03-23T14:36:53.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.763+0000 7f00d8ff9640 1 -- 172.21.15.161:0/200332621 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f00d0004b80 con 0x7f00dc0d1470 2024-03-23T14:36:53.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.763+0000 7f00d8ff9640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f00c806b810 0x7f00c806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:53.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.763+0000 7f00d8ff9640 1 -- 172.21.15.161:0/200332621 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4943+0+0 (secure 0 0 0) 0x7f00d0089400 con 0x7f00dc0d1470 2024-03-23T14:36:53.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.763+0000 7f00db7fe640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f00c806b810 0x7f00c806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:53.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.764+0000 7f00db7fe640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f00c806b810 0x7f00c806dcd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0b5490 tx=0x7f00cc006be0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:53.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.767+0000 7f00d8ff9640 1 -- 172.21.15.161:0/200332621 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f00d00541b0 con 0x7f00dc0d1470 2024-03-23T14:36:53.970 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.974+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00dc1442c0 con 0x7f00dc0d1470 2024-03-23T14:36:53.971 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.975+0000 7f00d8ff9640 1 -- 172.21.15.161:0/200332621 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 18 v18) ==== 94+0+3868 (secure 0 0 0) 0x7f00d0057870 con 0x7f00dc0d1470 2024-03-23T14:36:53.971 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:53.971 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":18,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4922,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/626265302","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":626265302},{"type":"v1","addr":"172.21.15.183:6837","nonce":626265302}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5220,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6833/208344896","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":208344896},{"type":"v1","addr":"172.21.15.161:6833","nonce":208344896}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":5223,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/82303731","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6834","nonce":82303731},{"type":"v1","addr":"172.21.15.161:6835","nonce":82303731}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":5779,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2000032444","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2000032444},{"type":"v1","addr":"172.21.15.183:6836","nonce":2000032444}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":18}],"filesystems":[]} 2024-03-23T14:36:53.971 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 18 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f00c806b810 msgr2=0x7f00c806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f00c806b810 0x7f00c806dcd0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f00dc0b5490 tx=0x7f00cc006be0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d1470 msgr2=0x7f00dc1445f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d1470 0x7f00dc1445f0 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f00d000c4d0 tx=0x7f00d000c990 comp rx=0 tx=0).stop 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 shutdown_connections 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f00c806b810 0x7f00c806dcd0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f00dc15e400 0x7f00dc0c9b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f00dc0d2b40 0x7f00dc15c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.973 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 --2- 172.21.15.161:0/200332621 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f00dc0d1470 0x7f00dc1445f0 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:53.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 >> 172.21.15.161:0/200332621 conn(0x7f00dc09d190 msgr2=0x7f00dc0d2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:53.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 shutdown_connections 2024-03-23T14:36:53.974 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:53.977+0000 7f00e260e640 1 -- 172.21.15.161:0/200332621 wait complete. 2024-03-23T14:36:53.984 INFO:tasks.ceph.mds.a:Restarting daemon 2024-03-23T14:36:53.984 INFO:tasks.ceph.mds.a:Stopping old one... 2024-03-23T14:36:53.984 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T14:36:53.984 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:36:53.985 INFO:tasks.ceph.mds.c:Restarting daemon 2024-03-23T14:36:53.985 INFO:tasks.ceph.mds.c:Stopping old one... 2024-03-23T14:36:53.985 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T14:36:53.985 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:36:53.985 INFO:tasks.ceph.mds.b:Restarting daemon 2024-03-23T14:36:53.985 INFO:tasks.ceph.mds.b:Stopping old one... 2024-03-23T14:36:53.985 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T14:36:53.986 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:36:53.986 INFO:tasks.ceph.mds.d:Restarting daemon 2024-03-23T14:36:53.986 INFO:tasks.ceph.mds.d:Stopping old one... 2024-03-23T14:36:53.986 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T14:36:53.986 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:36:53.994 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T14:36:53.994 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-03-23T14:36:53.995 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T14:36:53.995 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-03-23T14:36:53.999 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T14:36:53.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-03-23T14:36:54.001 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T14:36:54.001 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-03-23T14:36:54.036 INFO:tasks.ceph.mds.a:Started 2024-03-23T14:36:54.037 DEBUG:teuthology.parallel:result is None 2024-03-23T14:36:54.038 INFO:tasks.ceph.mds.c:Started 2024-03-23T14:36:54.038 DEBUG:teuthology.parallel:result is None 2024-03-23T14:36:54.042 INFO:tasks.ceph.mds.b:Started 2024-03-23T14:36:54.042 DEBUG:teuthology.parallel:result is None 2024-03-23T14:36:54.043 INFO:tasks.ceph.mds.d:Started 2024-03-23T14:36:54.044 DEBUG:teuthology.parallel:result is None 2024-03-23T14:36:54.044 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist clear 2024-03-23T14:36:54.113 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-23T14:36:54.113 INFO:tasks.ceph.mds.c.smithi161.stdout:starting mds.c at 2024-03-23T14:36:54.127 INFO:tasks.ceph.mds.b.smithi183.stdout:starting mds.b at 2024-03-23T14:36:54.127 INFO:tasks.ceph.mds.d.smithi183.stdout:starting mds.d at 2024-03-23T14:36:54.127 INFO:tasks.ceph.mds.d.smithi183.stderr:2024-03-23T14:36:54.132+0000 7fcadd809640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6834/0: (98) Address already in use 2024-03-23T14:36:54.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.156+0000 7f67df455640 1 Processor -- start 2024-03-23T14:36:54.152 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.156+0000 7f67df455640 1 -- start start 2024-03-23T14:36:54.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67df455640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f67d80d2b40 0x7f67d80b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:54.155 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67df455640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d80d0460 con 0x7f67d80d1470 2024-03-23T14:36:54.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67df455640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67d80d06b0 con 0x7f67d815e400 2024-03-23T14:36:54.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67df455640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f67d80d1050 con 0x7f67d80d2b40 2024-03-23T14:36:54.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67dc9f0640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f67d80d1470 0x7f67d80d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48500/0 (socket says 172.21.15.161:48500) 2024-03-23T14:36:54.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67dc9f0640 1 -- 172.21.15.161:0/1524669936 learned_addr learned my addr 172.21.15.161:0/1524669936 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:54.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67dd9f2640 1 --2- 172.21.15.161:0/1524669936 >> v2:172.21.15.183:3300/0 conn(0x7f67d80d2b40 0x7f67d80b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:54.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67cffff640 1 -- 172.21.15.161:0/1524669936 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401779813 0 0) 0x7f67d80d0460 con 0x7f67d80d1470 2024-03-23T14:36:54.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67cffff640 1 -- 172.21.15.161:0/1524669936 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67c4003520 con 0x7f67d80d1470 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67dd9f2640 1 -- 172.21.15.161:0/1524669936 >> v1:172.21.15.183:6789/0 conn(0x7f67d815e400 legacy=0x7f67d80cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67dd9f2640 1 -- 172.21.15.161:0/1524669936 >> v1:172.21.15.161:6789/0 conn(0x7f67d80d1470 legacy=0x7f67d80d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.157+0000 7f67dd9f2640 1 -- 172.21.15.161:0/1524669936 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d80d0990 con 0x7f67d80d2b40 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.158+0000 7f67dd9f2640 1 --2- 172.21.15.161:0/1524669936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d2b40 0x7f67d80b0a80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009d90 tx=0x7f67d002bf50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fafbe1b871547c6a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.158+0000 7f67cffff640 1 -- 172.21.15.161:0/1524669936 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d002cd00 con 0x7f67d80d2b40 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.158+0000 7f67cffff640 1 -- 172.21.15.161:0/1524669936 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d002ce60 con 0x7f67d80d2b40 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.158+0000 7f67cffff640 1 -- 172.21.15.161:0/1524669936 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d0034660 con 0x7f67d80d2b40 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.159+0000 7f67df455640 1 -- 172.21.15.161:0/1524669936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d2b40 msgr2=0x7f67d80b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:54.157 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.159+0000 7f67df455640 1 --2- 172.21.15.161:0/1524669936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d2b40 0x7f67d80b0a80 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f67d0009d90 tx=0x7f67d002bf50 comp rx=0 tx=0).stop 2024-03-23T14:36:54.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.159+0000 7f67df455640 1 -- 172.21.15.161:0/1524669936 shutdown_connections 2024-03-23T14:36:54.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.159+0000 7f67df455640 1 --2- 172.21.15.161:0/1524669936 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d2b40 0x7f67d80b0a80 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:54.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.159+0000 7f67df455640 1 -- 172.21.15.161:0/1524669936 >> 172.21.15.161:0/1524669936 conn(0x7f67d809d190 msgr2=0x7f67d80d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:54.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.159+0000 7f67df455640 1 -- 172.21.15.161:0/1524669936 shutdown_connections 2024-03-23T14:36:54.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.159+0000 7f67df455640 1 -- 172.21.15.161:0/1524669936 wait complete. 2024-03-23T14:36:54.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.160+0000 7f67df455640 1 Processor -- start 2024-03-23T14:36:54.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.160+0000 7f67df455640 1 -- start start 2024-03-23T14:36:54.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67df455640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d1470 0x7f67d81f3660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67df455640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d80d2b40 0x7f67d815b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67df455640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d815e400 0x7f67d815b720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67df455640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f67d81577f0 con 0x7f67d80d2b40 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67df455640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f67d8156bc0 con 0x7f67d80d1470 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67df455640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f67d8156e90 con 0x7f67d815e400 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dc9f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d80d2b40 0x7f67d815b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dc9f0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d80d2b40 0x7f67d815b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45000/0 (socket says 172.21.15.161:45000) 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dc9f0640 1 -- 172.21.15.161:0/4124676453 learned_addr learned my addr 172.21.15.161:0/4124676453 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dd1f1640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d1470 0x7f67d81f3660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dd9f2640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d815e400 0x7f67d815b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:54.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dc9f0640 1 -- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d815e400 msgr2=0x7f67d815b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dc9f0640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d815e400 0x7f67d815b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dc9f0640 1 -- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d1470 msgr2=0x7f67d81f3660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dc9f0640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d1470 0x7f67d81f3660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.161+0000 7f67dc9f0640 1 -- 172.21.15.161:0/4124676453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67d0009a70 con 0x7f67d80d2b40 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.162+0000 7f67dc9f0640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d80d2b40 0x7f67d815b1e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f67d400c410 tx=0x7f67d400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.162+0000 7f67cdffb640 1 -- 172.21.15.161:0/4124676453 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d4005ce0 con 0x7f67d80d2b40 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.162+0000 7f67cdffb640 1 -- 172.21.15.161:0/4124676453 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f67d4005e40 con 0x7f67d80d2b40 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.162+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67d8157130 con 0x7f67d80d2b40 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.162+0000 7f67cdffb640 1 -- 172.21.15.161:0/4124676453 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d40072f0 con 0x7f67d80d2b40 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.162+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67d81601e0 con 0x7f67d80d2b40 2024-03-23T14:36:54.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.163+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67d80a1e00 con 0x7f67d80d2b40 2024-03-23T14:36:54.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.163+0000 7f67cdffb640 1 -- 172.21.15.161:0/4124676453 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f67d4007510 con 0x7f67d80d2b40 2024-03-23T14:36:54.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.164+0000 7f67cdffb640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f67bc06b810 0x7f67bc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:54.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.164+0000 7f67cdffb640 1 -- 172.21.15.161:0/4124676453 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(41..41 src has 1..41) ==== 4943+0+0 (secure 0 0 0) 0x7f67d40893f0 con 0x7f67d80d2b40 2024-03-23T14:36:54.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.164+0000 7f67dd1f1640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f67bc06b810 0x7f67bc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:54.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.164+0000 7f67dd1f1640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f67bc06b810 0x7f67bc06dcd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000f60 tx=0x7f67c8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:54.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.168+0000 7f67cdffb640 1 -- 172.21.15.161:0/4124676453 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f67d4054310 con 0x7f67d80d2b40 2024-03-23T14:36:54.350 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:54.354+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist clear"} v 0) -- 0x7f67d80b9a30 con 0x7f67d80d2b40 2024-03-23T14:36:55.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.135+0000 7f67cdffb640 1 -- 172.21.15.161:0/4124676453 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist clear"}]=0 removed all blocklist entries v43) ==== 97+0+0 (secure 0 0 0) 0x7f67d40579d0 con 0x7f67d80d2b40 2024-03-23T14:36:55.131 INFO:teuthology.orchestra.run.smithi161.stderr: removed all blocklist entries 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f67bc06b810 msgr2=0x7f67bc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f67bc06b810 0x7f67bc06dcd0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f67c8000f60 tx=0x7f67c8009040 comp rx=0 tx=0).stop 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d80d2b40 msgr2=0x7f67d815b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d80d2b40 0x7f67d815b1e0 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f67d400c410 tx=0x7f67d400c8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 shutdown_connections 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f67bc06b810 0x7f67bc06dcd0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f67d815e400 0x7f67d815b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f67d80d2b40 0x7f67d815b1e0 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 --2- 172.21.15.161:0/4124676453 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f67d80d1470 0x7f67d81f3660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 >> 172.21.15.161:0/4124676453 conn(0x7f67d809d190 msgr2=0x7f67d815d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:55.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 shutdown_connections 2024-03-23T14:36:55.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.139+0000 7f67df455640 1 -- 172.21.15.161:0/4124676453 wait complete. 2024-03-23T14:36:55.148 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2024-03-23T14:36:55.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.254+0000 7f09afb2e640 1 Processor -- start 2024-03-23T14:36:55.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.254+0000 7f09afb2e640 1 -- start start 2024-03-23T14:36:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.255+0000 7f09afb2e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09a815e400 0x7f09a80cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.255+0000 7f09afb2e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09a80cf9d0 0x7f09a80d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.255+0000 7f09afb2e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09a80d0460 con 0x7f09a80d1470 2024-03-23T14:36:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.255+0000 7f09afb2e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f09a80d1050 con 0x7f09a815e400 2024-03-23T14:36:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.255+0000 7f09afb2e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f09a80d06b0 con 0x7f09a80cf9d0 2024-03-23T14:36:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.255+0000 7f09ad8ca640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f09a815e400 0x7f09a80cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.255+0000 7f09ae0cb640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f09a80d1470 0x7f09a80b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48506/0 (socket says 172.21.15.161:48506) 2024-03-23T14:36:55.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.255+0000 7f09ae0cb640 1 -- 172.21.15.161:0/3626641834 learned_addr learned my addr 172.21.15.161:0/3626641834 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.256+0000 7f09ad8ca640 1 -- 172.21.15.161:0/3626641834 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09a80cf9d0 msgr2=0x7f09a80d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.256+0000 7f09ad8ca640 1 --2- 172.21.15.161:0/3626641834 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09a80cf9d0 0x7f09a80d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.256+0000 7f09ad8ca640 1 -- 172.21.15.161:0/3626641834 >> v1:172.21.15.161:6789/0 conn(0x7f09a80d1470 legacy=0x7f09a80b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.256+0000 7f09ad8ca640 1 -- 172.21.15.161:0/3626641834 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a80d0990 con 0x7f09a815e400 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.256+0000 7f09ac8c8640 1 -- 172.21.15.161:0/3626641834 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 822786239 0 0) 0x7f09a80d0460 con 0x7f09a80d1470 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.256+0000 7f09ad8ca640 1 --2- 172.21.15.161:0/3626641834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a815e400 0x7f09a80cf490 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f09a40097a0 tx=0x7f09a402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3830e0f5c13fa60c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.256+0000 7f09ac8c8640 1 -- 172.21.15.161:0/3626641834 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4031830 con 0x7f09a815e400 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09ac8c8640 1 -- 172.21.15.161:0/3626641834 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a4031990 con 0x7f09a815e400 2024-03-23T14:36:55.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09ac8c8640 1 -- 172.21.15.161:0/3626641834 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a4031d20 con 0x7f09a815e400 2024-03-23T14:36:55.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09afb2e640 1 -- 172.21.15.161:0/3626641834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a815e400 msgr2=0x7f09a80cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09afb2e640 1 --2- 172.21.15.161:0/3626641834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a815e400 0x7f09a80cf490 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f09a40097a0 tx=0x7f09a402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09afb2e640 1 -- 172.21.15.161:0/3626641834 shutdown_connections 2024-03-23T14:36:55.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09afb2e640 1 --2- 172.21.15.161:0/3626641834 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09a80cf9d0 0x7f09a80d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09afb2e640 1 --2- 172.21.15.161:0/3626641834 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a815e400 0x7f09a80cf490 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09afb2e640 1 -- 172.21.15.161:0/3626641834 >> 172.21.15.161:0/3626641834 conn(0x7f09a809d190 msgr2=0x7f09a80d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09afb2e640 1 -- 172.21.15.161:0/3626641834 shutdown_connections 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.257+0000 7f09afb2e640 1 -- 172.21.15.161:0/3626641834 wait complete. 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.258+0000 7f09afb2e640 1 Processor -- start 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.258+0000 7f09afb2e640 1 -- start start 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09afb2e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a80cf9d0 0x7f09a81f36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09afb2e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09a80d1470 0x7f09a80b5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09afb2e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a815e400 0x7f09a80c83b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09afb2e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f09a815dc30 con 0x7f09a80cf9d0 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09afb2e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f09a80b42d0 con 0x7f09a815e400 2024-03-23T14:36:55.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09afb2e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f09a80b45b0 con 0x7f09a80d1470 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a80cf9d0 0x7f09a81f36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a80cf9d0 0x7f09a81f36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45018/0 (socket says 172.21.15.161:45018) 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 -- 172.21.15.161:0/545916410 learned_addr learned my addr 172.21.15.161:0/545916410 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 -- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09a80d1470 msgr2=0x7f09a80b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09a80d1470 0x7f09a80b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 -- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a815e400 msgr2=0x7f09a80c83b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a815e400 0x7f09a80c83b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 -- 172.21.15.161:0/545916410 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09a4009400 con 0x7f09a80cf9d0 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.259+0000 7f09ad8ca640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a80cf9d0 0x7f09a81f36d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f09a4000c00 tx=0x7f09a40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.260+0000 7f099e7fc640 1 -- 172.21.15.161:0/545916410 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a403aa60 con 0x7f09a80cf9d0 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.260+0000 7f099e7fc640 1 -- 172.21.15.161:0/545916410 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a403abc0 con 0x7f09a80cf9d0 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.260+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09a80b4890 con 0x7f09a80cf9d0 2024-03-23T14:36:55.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.260+0000 7f099e7fc640 1 -- 172.21.15.161:0/545916410 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a40443f0 con 0x7f09a80cf9d0 2024-03-23T14:36:55.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.260+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09a80b4e50 con 0x7f09a80cf9d0 2024-03-23T14:36:55.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.261+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09a80a1e00 con 0x7f09a80cf9d0 2024-03-23T14:36:55.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.261+0000 7f099e7fc640 1 -- 172.21.15.161:0/545916410 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f09a404d450 con 0x7f09a80cf9d0 2024-03-23T14:36:55.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.261+0000 7f099e7fc640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f098c06b840 0x7f098c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.262+0000 7f099e7fc640 1 -- 172.21.15.161:0/545916410 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4857+0+0 (secure 0 0 0) 0x7f09a40ac040 con 0x7f09a80cf9d0 2024-03-23T14:36:55.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.262+0000 7f09ad0c9640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f098c06b840 0x7f098c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.262+0000 7f09ad0c9640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f098c06b840 0x7f098c06dd00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f09a81f7090 tx=0x7f0998006c90 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.266+0000 7f099e7fc640 1 -- 172.21.15.161:0/545916410 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f09a4076e60 con 0x7f09a80cf9d0 2024-03-23T14:36:55.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.500+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) -- 0x7f09a804f550 con 0x7f09a80cf9d0 2024-03-23T14:36:55.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.500+0000 7f099e7fc640 1 -- 172.21.15.161:0/545916410 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v14) ==== 80+0+5298 (secure 0 0 0) 0x7f09a407a520 con 0x7f09a80cf9d0 2024-03-23T14:36:55.496 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: "auth_dump": [ 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.a", 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lzaKfFRAAc+l/nk//V/gpMXQ1JO3ATg==", 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.497 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.b", 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lcMUxIhAAA7sdubbg/sFxR2JSLj6knA==", 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.498 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.c", 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lOl8SHRAAPqXulseV9GlBMsoYeIUt5Q==", 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.499 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mds.d", 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5la0WTKRAACwJ5zIvawF/Gsu3pgRkjRQ==", 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.500 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.0", 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBl6P5lv7pCExAACgyFS3tWsnorh+23JfVMFQ==", 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.1", 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBo6P5lN9XjERAAY7Doq86QfXJ46LU3H9ET4g==", 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:55.501 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.2", 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBr6P5le9jdDxAAF7YY5jy4+03N9tB79FqgxA==", 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.502 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.3", 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBu6P5lfyVdDhAA6GSFuGvPYAUCczsnMpXFkQ==", 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.4", 2024-03-23T14:36:55.503 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQB06P5lxMy+FhAAhO1d5E1ioKlC3bqbkingpA==", 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.5", 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQB36P5lHl2nFRAA18JrgqJgj6mYykRjhemRvQ==", 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.504 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.6", 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQB66P5lC5OzFBAAeJslkQ08ZxK2fx9yQKsCOA==", 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.505 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "osd.7", 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQB96P5lyiCSEhAAn+5LSA94oOhlckTfTUd7Lw==", 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow profile osd", 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile osd", 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.0", 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g==", 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:55.506 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow r", 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rw pool=cephfs_data" 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.1", 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lskU4NxAAed0RZpsIuzbux+yi642hsw==", 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.507 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.2", 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lIYbHMxAAOYcn/3VZTDW2K3OBKfjxkA==", 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.3", 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBi6P5lwCRzARAAlFyun7fjX8hB/KsVJhLfWg==", 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.508 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow", 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow r", 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow rw", 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow rwx" 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "client.admin", 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBg6P5lHQT4MxAAvLp+7Sae/Py5kxSOJ51MzQ==", 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "mgr": "allow *", 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow *", 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.509 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.x", 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lz/YnEhAA4Q/N4Xnmj7rkBGQH0WFU5w==", 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "entity": "mgr.y", 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "key": "AQBh6P5lMbfQDRAA7IKJtMZ6Z77M6m4dD75Vcg==", 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "caps": { 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "mds": "allow *", 2024-03-23T14:36:55.510 INFO:teuthology.orchestra.run.smithi161.stdout: "mon": "allow profile mgr", 2024-03-23T14:36:55.511 INFO:teuthology.orchestra.run.smithi161.stdout: "osd": "allow *" 2024-03-23T14:36:55.511 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.511 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:55.511 INFO:teuthology.orchestra.run.smithi161.stdout: ] 2024-03-23T14:36:55.511 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:36:55.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.502+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f098c06b840 msgr2=0x7f098c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.502+0000 7f09afb2e640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f098c06b840 0x7f098c06dd00 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f09a81f7090 tx=0x7f0998006c90 comp rx=0 tx=0).stop 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.502+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a80cf9d0 msgr2=0x7f09a81f36d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.502+0000 7f09afb2e640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a80cf9d0 0x7f09a81f36d0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f09a4000c00 tx=0x7f09a40046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.502+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 shutdown_connections 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.502+0000 7f09afb2e640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f098c06b840 0x7f098c06dd00 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.503+0000 7f09afb2e640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f09a815e400 0x7f09a80c83b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.503+0000 7f09afb2e640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f09a80d1470 0x7f09a80b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.503+0000 7f09afb2e640 1 --2- 172.21.15.161:0/545916410 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f09a80cf9d0 0x7f09a81f36d0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.503+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 >> 172.21.15.161:0/545916410 conn(0x7f09a809d190 msgr2=0x7f09a806ce80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.503+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 shutdown_connections 2024-03-23T14:36:55.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.503+0000 7f09afb2e640 1 -- 172.21.15.161:0/545916410 wait complete. 2024-03-23T14:36:55.514 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.618+0000 7f2bef26f640 1 Processor -- start 2024-03-23T14:36:55.614 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.618+0000 7f2bef26f640 1 -- start start 2024-03-23T14:36:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bef26f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2be81601f0 0x7f2be8072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bef26f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be80cf4f0 con 0x7f2be81607d0 2024-03-23T14:36:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bef26f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be80cef30 con 0x7f2be8145080 2024-03-23T14:36:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bef26f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2be8071010 con 0x7f2be81601f0 2024-03-23T14:36:55.615 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bec80a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2be81601f0 0x7f2be8072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bec80a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2be81601f0 0x7f2be8072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45026/0 (socket says 172.21.15.161:45026) 2024-03-23T14:36:55.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bec80a640 1 -- 172.21.15.161:0/1419458293 learned_addr learned my addr 172.21.15.161:0/1419458293 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:55.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bed80c640 1 --1- 172.21.15.161:0/1419458293 >> v1:172.21.15.161:6789/0 conn(0x7f2be81607d0 0x7f2be815c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48508/0 (socket says 172.21.15.161:48508) 2024-03-23T14:36:55.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bec80a640 1 -- 172.21.15.161:0/1419458293 >> v1:172.21.15.183:6789/0 conn(0x7f2be8145080 legacy=0x7f2be815fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bec80a640 1 -- 172.21.15.161:0/1419458293 >> v1:172.21.15.161:6789/0 conn(0x7f2be81607d0 legacy=0x7f2be815c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bdffff640 1 -- 172.21.15.161:0/1419458293 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 267124732 0 0) 0x7f2be80cf4f0 con 0x7f2be81607d0 2024-03-23T14:36:55.616 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.619+0000 7f2bec80a640 1 -- 172.21.15.161:0/1419458293 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be80cf210 con 0x7f2be81601f0 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.620+0000 7f2bec80a640 1 --2- 172.21.15.161:0/1419458293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be8072a00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009870 tx=0x7f2bd802b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a93f0e0830837f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.620+0000 7f2bdffff640 1 -- 172.21.15.161:0/1419458293 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8030860 con 0x7f2be81601f0 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.620+0000 7f2bdffff640 1 -- 172.21.15.161:0/1419458293 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd80309c0 con 0x7f2be81601f0 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.620+0000 7f2bdffff640 1 -- 172.21.15.161:0/1419458293 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8030d50 con 0x7f2be81601f0 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.621+0000 7f2bef26f640 1 -- 172.21.15.161:0/1419458293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 msgr2=0x7f2be8072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.621+0000 7f2bef26f640 1 --2- 172.21.15.161:0/1419458293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be8072a00 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009870 tx=0x7f2bd802b9a0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.621+0000 7f2bef26f640 1 -- 172.21.15.161:0/1419458293 shutdown_connections 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.621+0000 7f2bef26f640 1 --2- 172.21.15.161:0/1419458293 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be8072a00 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.621+0000 7f2bef26f640 1 -- 172.21.15.161:0/1419458293 >> 172.21.15.161:0/1419458293 conn(0x7f2be80b17f0 msgr2=0x7f2be80b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.621+0000 7f2bef26f640 1 -- 172.21.15.161:0/1419458293 shutdown_connections 2024-03-23T14:36:55.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.621+0000 7f2bef26f640 1 -- 172.21.15.161:0/1419458293 wait complete. 2024-03-23T14:36:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.622+0000 7f2bef26f640 1 Processor -- start 2024-03-23T14:36:55.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.622+0000 7f2bef26f640 1 -- start start 2024-03-23T14:36:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bef26f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2be8145080 0x7f2be80b3b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bef26f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be80c4a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bef26f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2be81607d0 0x7f2be80c4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bef26f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80c54e0 con 0x7f2be81601f0 2024-03-23T14:36:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bef26f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2be80c56e0 con 0x7f2be81607d0 2024-03-23T14:36:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bef26f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2be80c59c0 con 0x7f2be8145080 2024-03-23T14:36:55.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be80c4a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be80c4a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45032/0 (socket says 172.21.15.161:45032) 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 -- 172.21.15.161:0/2267165248 learned_addr learned my addr 172.21.15.161:0/2267165248 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bed00b640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2be8145080 0x7f2be80b3b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bed80c640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2be81607d0 0x7f2be80c4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 -- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2be8145080 msgr2=0x7f2be80b3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2be8145080 0x7f2be80b3b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 -- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2be81607d0 msgr2=0x7f2be80c4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2be81607d0 0x7f2be80c4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 -- 172.21.15.161:0/2267165248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bd80094d0 con 0x7f2be81601f0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.623+0000 7f2bec80a640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be80c4a60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009870 tx=0x7f2bd8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.624+0000 7f2bddffb640 1 -- 172.21.15.161:0/2267165248 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd8030810 con 0x7f2be81601f0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.624+0000 7f2bddffb640 1 -- 172.21.15.161:0/2267165248 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2bd8043400 con 0x7f2be81601f0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.624+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2be80c5ca0 con 0x7f2be81601f0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.624+0000 7f2bddffb640 1 -- 172.21.15.161:0/2267165248 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bd80437b0 con 0x7f2be81601f0 2024-03-23T14:36:55.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.624+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2be80c6260 con 0x7f2be81601f0 2024-03-23T14:36:55.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.625+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2be806cb70 con 0x7f2be81601f0 2024-03-23T14:36:55.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.625+0000 7f2bddffb640 1 -- 172.21.15.161:0/2267165248 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2bd8043910 con 0x7f2be81601f0 2024-03-23T14:36:55.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.626+0000 7f2bddffb640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2bcc06b810 0x7f2bcc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.626+0000 7f2bddffb640 1 -- 172.21.15.161:0/2267165248 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4857+0+0 (secure 0 0 0) 0x7f2bd803e070 con 0x7f2be81601f0 2024-03-23T14:36:55.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.626+0000 7f2bed00b640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2bcc06b810 0x7f2bcc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.626+0000 7f2bed00b640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2bcc06b810 0x7f2bcc06dcd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2be4000f60 tx=0x7f2be4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.625 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.630+0000 7f2bddffb640 1 -- 172.21.15.161:0/2267165248 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2bd8078020 con 0x7f2be81601f0 2024-03-23T14:36:55.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.802+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2be80735e0 con 0x7f2be81601f0 2024-03-23T14:36:55.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.803+0000 7f2bddffb640 1 -- 172.21.15.161:0/2267165248 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v44) ==== 74+0+12927 (secure 0 0 0) 0x7f2bd807b6e0 con 0x7f2be81601f0 2024-03-23T14:36:55.799 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:55.799 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":44,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:55.139341+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2bcc06b810 msgr2=0x7f2bcc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2bcc06b810 0x7f2bcc06dcd0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2be4000f60 tx=0x7f2be4009040 comp rx=0 tx=0).stop 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 msgr2=0x7f2be80c4a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be80c4a60 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f2bd8009870 tx=0x7f2bd8004730 comp rx=0 tx=0).stop 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 shutdown_connections 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2bcc06b810 0x7f2bcc06dcd0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2be81607d0 0x7f2be80c4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2be81601f0 0x7f2be80c4a60 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 --2- 172.21.15.161:0/2267165248 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2be8145080 0x7f2be80b3b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 >> 172.21.15.161:0/2267165248 conn(0x7f2be80b17f0 msgr2=0x7f2be80b0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:55.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 shutdown_connections 2024-03-23T14:36:55.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.805+0000 7f2bef26f640 1 -- 172.21.15.161:0/2267165248 wait complete. 2024-03-23T14:36:55.812 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:36:55.813 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:36:55.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.919+0000 7f3d322f1640 1 Processor -- start 2024-03-23T14:36:55.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.920+0000 7f3d322f1640 1 -- start start 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.920+0000 7f3d322f1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3d2c15e470 0x7f3d2c0d45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.920+0000 7f3d322f1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c0d24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.920+0000 7f3d322f1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2c0cf800 con 0x7f3d2c15e850 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.920+0000 7f3d322f1640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f3d2c0d03a0 con 0x7f3d2c15e470 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.920+0000 7f3d322f1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2c0ceed0 con 0x7f3d2c0b5510 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d3088e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c0d24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d2b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f3d2c15e850 0x7f3d2c0cfc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:54788/0 (socket says 172.21.15.161:54788) 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d3088e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c0d24b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:51470/0 (socket says 172.21.15.161:51470) 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d2bfff640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3d2c15e470 0x7f3d2c0d45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d2b7fe640 1 -- 172.21.15.161:0/3057795319 learned_addr learned my addr 172.21.15.161:0/3057795319 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d2affd640 1 -- 172.21.15.161:0/3057795319 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2128239489 0 0) 0x7f3d2c0cf800 con 0x7f3d2c15e850 2024-03-23T14:36:55.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d2affd640 1 -- 172.21.15.161:0/3057795319 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d14003520 con 0x7f3d2c15e850 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d3088e640 1 -- 172.21.15.161:0/3057795319 >> v2:172.21.15.183:3300/0 conn(0x7f3d2c15e470 msgr2=0x7f3d2c0d45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d3088e640 1 --2- 172.21.15.161:0/3057795319 >> v2:172.21.15.183:3300/0 conn(0x7f3d2c15e470 0x7f3d2c0d45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d3088e640 1 -- 172.21.15.161:0/3057795319 >> v1:172.21.15.183:6789/0 conn(0x7f3d2c15e850 legacy=0x7f3d2c0cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.921+0000 7f3d3088e640 1 -- 172.21.15.161:0/3057795319 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2c0cf1b0 con 0x7f3d2c0b5510 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.922+0000 7f3d3088e640 1 --2- 172.21.15.161:0/3057795319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c0d24b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009d90 tx=0x7f3d2402be10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1e6e52e8c0519338 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.922+0000 7f3d2affd640 1 -- 172.21.15.161:0/3057795319 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d2402cbc0 con 0x7f3d2c0b5510 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.922+0000 7f3d2affd640 1 -- 172.21.15.161:0/3057795319 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d2402cd20 con 0x7f3d2c0b5510 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.922+0000 7f3d2affd640 1 -- 172.21.15.161:0/3057795319 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24034640 con 0x7f3d2c0b5510 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.923+0000 7f3d322f1640 1 -- 172.21.15.161:0/3057795319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 msgr2=0x7f3d2c0d24b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.923+0000 7f3d322f1640 1 --2- 172.21.15.161:0/3057795319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c0d24b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009d90 tx=0x7f3d2402be10 comp rx=0 tx=0).stop 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.923+0000 7f3d322f1640 1 -- 172.21.15.161:0/3057795319 shutdown_connections 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.923+0000 7f3d322f1640 1 --2- 172.21.15.161:0/3057795319 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c0d24b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.923+0000 7f3d322f1640 1 --2- 172.21.15.161:0/3057795319 >> v2:172.21.15.183:3300/0 conn(0x7f3d2c15e470 0x7f3d2c0d45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.923+0000 7f3d322f1640 1 -- 172.21.15.161:0/3057795319 >> 172.21.15.161:0/3057795319 conn(0x7f3d2c09d180 msgr2=0x7f3d2c0d3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:55.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.923+0000 7f3d322f1640 1 -- 172.21.15.161:0/3057795319 shutdown_connections 2024-03-23T14:36:55.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.923+0000 7f3d322f1640 1 -- 172.21.15.161:0/3057795319 wait complete. 2024-03-23T14:36:55.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.924+0000 7f3d322f1640 1 Processor -- start 2024-03-23T14:36:55.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.924+0000 7f3d322f1640 1 -- start start 2024-03-23T14:36:55.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d322f1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c150320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d322f1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d2c15e470 0x7f3d2c0cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d322f1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d2c15e850 0x7f3d2c06b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d322f1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c0c7d40 con 0x7f3d2c15e470 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d322f1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2c0ce310 con 0x7f3d2c15e850 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d322f1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2c0ca3e0 con 0x7f3d2c0b5510 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d2c15e470 0x7f3d2c0cddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d2c15e470 0x7f3d2c0cddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45040/0 (socket says 172.21.15.161:45040) 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 -- 172.21.15.161:0/4145120514 learned_addr learned my addr 172.21.15.161:0/4145120514 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d3088e640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d2c15e850 0x7f3d2c06b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2bfff640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c150320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 -- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 msgr2=0x7f3d2c150320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c150320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 -- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d2c15e850 msgr2=0x7f3d2c06b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d2c15e850 0x7f3d2c06b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 -- 172.21.15.161:0/4145120514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d24009a70 con 0x7f3d2c15e470 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2bfff640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:55.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d2b7fe640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d2c15e470 0x7f3d2c0cddd0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000c430 tx=0x7f3d2000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d28ff9640 1 -- 172.21.15.161:0/4145120514 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20005ce0 con 0x7f3d2c15e470 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d28ff9640 1 -- 172.21.15.161:0/4145120514 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d20005e40 con 0x7f3d2c15e470 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d28ff9640 1 -- 172.21.15.161:0/4145120514 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200072f0 con 0x7f3d2c15e470 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.925+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d2c0ca6c0 con 0x7f3d2c15e470 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.926+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2c0cb550 con 0x7f3d2c15e470 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.927+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d2c04f4e0 con 0x7f3d2c15e470 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.927+0000 7f3d28ff9640 1 -- 172.21.15.161:0/4145120514 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3d20007450 con 0x7f3d2c15e470 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.927+0000 7f3d28ff9640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3d1806b810 0x7f3d1806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.927+0000 7f3d28ff9640 1 -- 172.21.15.161:0/4145120514 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4857+0+0 (secure 0 0 0) 0x7f3d200894c0 con 0x7f3d2c15e470 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.928+0000 7f3d2bfff640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3d1806b810 0x7f3d1806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:55.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.928+0000 7f3d2bfff640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3d1806b810 0x7f3d1806dcd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009ac0 tx=0x7f3d1c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:55.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:55.932+0000 7f3d28ff9640 1 -- 172.21.15.161:0/4145120514 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3d200542c0 con 0x7f3d2c15e470 2024-03-23T14:36:56.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.105+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3d2c09d6d0 con 0x7f3d2c15e470 2024-03-23T14:36:56.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.105+0000 7f3d28ff9640 1 -- 172.21.15.161:0/4145120514 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v44) ==== 93+0+10 (secure 0 0 0) 0x7f3d200170a0 con 0x7f3d2c15e470 2024-03-23T14:36:56.101 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:36:56.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3d1806b810 msgr2=0x7f3d1806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3d1806b810 0x7f3d1806dcd0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009ac0 tx=0x7f3d1c008040 comp rx=0 tx=0).stop 2024-03-23T14:36:56.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d2c15e470 msgr2=0x7f3d2c0cddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d2c15e470 0x7f3d2c0cddd0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000c430 tx=0x7f3d2000c8f0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 shutdown_connections 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3d1806b810 0x7f3d1806dcd0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3d2c15e850 0x7f3d2c06b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3d2c15e470 0x7f3d2c0cddd0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 --2- 172.21.15.161:0/4145120514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3d2c0b5510 0x7f3d2c150320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 >> 172.21.15.161:0/4145120514 conn(0x7f3d2c09d180 msgr2=0x7f3d2c0d40b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 shutdown_connections 2024-03-23T14:36:56.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.107+0000 7f3d322f1640 1 -- 172.21.15.161:0/4145120514 wait complete. 2024-03-23T14:36:56.114 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-23T14:36:56.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.220+0000 7ff4fb03b640 1 Processor -- start 2024-03-23T14:36:56.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.220+0000 7ff4fb03b640 1 -- start start 2024-03-23T14:36:56.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4fb03b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f40d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4fb03b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4f40d1270 0x7ff4f40d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4fb03b640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7ff4f40d29d0 0x7ff4f4161e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4fb03b640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7ff4f4161c00 con 0x7ff4f40d1270 2024-03-23T14:36:56.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4fb03b640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7ff4f40d2e20 con 0x7ff4f40d29d0 2024-03-23T14:36:56.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4fb03b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f40cea30 con 0x7ff4f4161820 2024-03-23T14:36:56.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4f40d1270 0x7ff4f40d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7ff4f40d1270 0x7ff4f40d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45048/0 (socket says 172.21.15.161:45048) 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f3fff640 1 -- 172.21.15.161:0/1245750468 learned_addr learned my addr 172.21.15.161:0/1245750468 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f95d8640 1 --2- 172.21.15.161:0/1245750468 >> v2:172.21.15.183:3300/0 conn(0x7ff4f40d29d0 0x7ff4f4161e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f8dd7640 1 --2- 172.21.15.161:0/1245750468 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f40d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f3fff640 1 -- 172.21.15.161:0/1245750468 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 msgr2=0x7ff4f40d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f3fff640 1 --2- 172.21.15.161:0/1245750468 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f40d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f3fff640 1 -- 172.21.15.161:0/1245750468 >> v2:172.21.15.183:3300/0 conn(0x7ff4f40d29d0 msgr2=0x7ff4f4161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f3fff640 1 --2- 172.21.15.161:0/1245750468 >> v2:172.21.15.183:3300/0 conn(0x7ff4f40d29d0 0x7ff4f4161e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f3fff640 1 -- 172.21.15.161:0/1245750468 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4f40cfd70 con 0x7ff4f40d1270 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f8dd7640 1 --2- 172.21.15.161:0/1245750468 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f40d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.221+0000 7ff4f95d8640 1 --2- 172.21.15.161:0/1245750468 >> v2:172.21.15.183:3300/0 conn(0x7ff4f40d29d0 0x7ff4f4161e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.222+0000 7ff4f3fff640 1 --2- 172.21.15.161:0/1245750468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d1270 0x7ff4f40d2490 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff4f409d630 tx=0x7ff4e802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81da52e4c189344a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:56.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.222+0000 7ff4f37fe640 1 -- 172.21.15.161:0/1245750468 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8031870 con 0x7ff4f40d1270 2024-03-23T14:36:56.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.222+0000 7ff4f37fe640 1 -- 172.21.15.161:0/1245750468 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e80319d0 con 0x7ff4f40d1270 2024-03-23T14:36:56.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.222+0000 7ff4f37fe640 1 -- 172.21.15.161:0/1245750468 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e8031d60 con 0x7ff4f40d1270 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 -- 172.21.15.161:0/1245750468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d1270 msgr2=0x7ff4f40d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/1245750468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d1270 0x7ff4f40d2490 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7ff4f409d630 tx=0x7ff4e802b910 comp rx=0 tx=0).stop 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 -- 172.21.15.161:0/1245750468 shutdown_connections 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/1245750468 >> v2:172.21.15.183:3300/0 conn(0x7ff4f40d29d0 0x7ff4f4161e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/1245750468 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d1270 0x7ff4f40d2490 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/1245750468 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f40d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 -- 172.21.15.161:0/1245750468 >> 172.21.15.161:0/1245750468 conn(0x7ff4f409d190 msgr2=0x7ff4f40d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 -- 172.21.15.161:0/1245750468 shutdown_connections 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.223+0000 7ff4fb03b640 1 -- 172.21.15.161:0/1245750468 wait complete. 2024-03-23T14:36:56.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.224+0000 7ff4fb03b640 1 Processor -- start 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.224+0000 7ff4fb03b640 1 -- start start 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4fb03b640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4f40d1270 0x7ff4f414a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4fb03b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d29d0 0x7ff4f414af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4fb03b640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f414d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4fb03b640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f40c9a40 con 0x7ff4f40d29d0 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4fb03b640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff4f40c9d20 con 0x7ff4f40d1270 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4fb03b640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff4f40ca000 con 0x7ff4f4161820 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d29d0 0x7ff4f414af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d29d0 0x7ff4f414af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45050/0 (socket says 172.21.15.161:45050) 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 -- 172.21.15.161:0/411673075 learned_addr learned my addr 172.21.15.161:0/411673075 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f95d8640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f414d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f8dd7640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4f40d1270 0x7ff4f414a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 -- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 msgr2=0x7ff4f414d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f414d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 -- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4f40d1270 msgr2=0x7ff4f414a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4f40d1270 0x7ff4f414a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 -- 172.21.15.161:0/411673075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4e80094d0 con 0x7ff4f40d29d0 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f95d8640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f414d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f8dd7640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4f40d1270 0x7ff4f414a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.225+0000 7ff4f3fff640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d29d0 0x7ff4f414af10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff4e802b950 tx=0x7ff4e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.226+0000 7ff4f17fa640 1 -- 172.21.15.161:0/411673075 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e803ad30 con 0x7ff4f40d29d0 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.226+0000 7ff4f17fa640 1 -- 172.21.15.161:0/411673075 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff4e80443f0 con 0x7ff4f40d29d0 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.226+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4f40ca2e0 con 0x7ff4f40d29d0 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.226+0000 7ff4f17fa640 1 -- 172.21.15.161:0/411673075 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4e80447a0 con 0x7ff4f40d29d0 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.226+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4f414db90 con 0x7ff4f40d29d0 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.227+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4f40ca470 con 0x7ff4f40d29d0 2024-03-23T14:36:56.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.227+0000 7ff4f17fa640 1 -- 172.21.15.161:0/411673075 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff4e804d450 con 0x7ff4f40d29d0 2024-03-23T14:36:56.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.228+0000 7ff4f17fa640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4e006b810 0x7ff4e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.228+0000 7ff4f17fa640 1 -- 172.21.15.161:0/411673075 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4857+0+0 (secure 0 0 0) 0x7ff4e802d040 con 0x7ff4f40d29d0 2024-03-23T14:36:56.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.228+0000 7ff4f8dd7640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4e006b810 0x7ff4e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.228+0000 7ff4f8dd7640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4e006b810 0x7ff4e006dcd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4006fd0 tx=0x7ff4e4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:56.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.232+0000 7ff4f17fa640 1 -- 172.21.15.161:0/411673075 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff4f40ca470 con 0x7ff4f40d29d0 2024-03-23T14:36:56.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.405+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7ff4f40ca470 con 0x7ff4f40d29d0 2024-03-23T14:36:56.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.405+0000 7ff4f17fa640 1 -- 172.21.15.161:0/411673075 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v44) ==== 84+0+65 (secure 0 0 0) 0x7ff4f40ca470 con 0x7ff4f40d29d0 2024-03-23T14:36:56.403 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:56.403 INFO:teuthology.orchestra.run.smithi161.stdout:[ 2024-03-23T14:36:56.403 INFO:teuthology.orchestra.run.smithi161.stdout: { 2024-03-23T14:36:56.403 INFO:teuthology.orchestra.run.smithi161.stdout: "poolnum": 1, 2024-03-23T14:36:56.403 INFO:teuthology.orchestra.run.smithi161.stdout: "poolname": ".mgr" 2024-03-23T14:36:56.403 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:36:56.403 INFO:teuthology.orchestra.run.smithi161.stdout:] 2024-03-23T14:36:56.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4e006b810 msgr2=0x7ff4e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4e006b810 0x7ff4e006dcd0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7ff4e4006fd0 tx=0x7ff4e4008040 comp rx=0 tx=0).stop 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d29d0 msgr2=0x7ff4f414af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d29d0 0x7ff4f414af10 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7ff4e802b950 tx=0x7ff4e80046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 shutdown_connections 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff4e006b810 0x7ff4e006dcd0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff4f4161820 0x7ff4f414d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff4f40d29d0 0x7ff4f414af10 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 --2- 172.21.15.161:0/411673075 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff4f40d1270 0x7ff4f414a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 >> 172.21.15.161:0/411673075 conn(0x7ff4f409d190 msgr2=0x7ff4f41622e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 shutdown_connections 2024-03-23T14:36:56.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.407+0000 7ff4fb03b640 1 -- 172.21.15.161:0/411673075 wait complete. 2024-03-23T14:36:56.419 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-23T14:36:56.419 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-23T14:36:56.537 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.541+0000 7f5e4c5d7640 1 Processor -- start 2024-03-23T14:36:56.538 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.541+0000 7f5e4c5d7640 1 -- start start 2024-03-23T14:36:56.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e4c5d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e440cf3b0 0x7f5e440cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e4c5d7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e440d03f0 con 0x7f5e440cfa50 2024-03-23T14:36:56.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e4c5d7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5e44144e80 con 0x7f5e44144aa0 2024-03-23T14:36:56.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e4c5d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5e440712a0 con 0x7f5e440cf3b0 2024-03-23T14:36:56.547 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e49b72640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5e440cfa50 0x7f5e440ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48510/0 (socket says 172.21.15.161:48510) 2024-03-23T14:36:56.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e49b72640 1 -- 172.21.15.161:0/2473013083 learned_addr learned my addr 172.21.15.161:0/2473013083 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:56.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e4ab74640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e440cf3b0 0x7f5e440cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 819437829 0 0) 0x7f5e440d03f0 con 0x7f5e440cfa50 2024-03-23T14:36:56.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.542+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e30003520 con 0x7f5e440cfa50 2024-03-23T14:36:56.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622055511 0 0) 0x7f5e44144e80 con 0x7f5e44144aa0 2024-03-23T14:36:56.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e440d03f0 con 0x7f5e44144aa0 2024-03-23T14:36:56.548 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3100001047 0 0) 0x7f5e30003520 con 0x7f5e440cfa50 2024-03-23T14:36:56.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e440cf3b0 msgr2=0x7f5e440cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:56.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e440cf3b0 0x7f5e440cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 >> v1:172.21.15.183:6789/0 conn(0x7f5e44144aa0 legacy=0x7f5e44070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e440b3800 con 0x7f5e440cfa50 2024-03-23T14:36:56.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f5e34001670 con 0x7f5e440cfa50 2024-03-23T14:36:56.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 >> v1:172.21.15.161:6789/0 conn(0x7f5e440cfa50 legacy=0x7f5e440ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e440cf3b0 0x7f5e30003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e300047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e30004d00 0x7f5e30005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5e440712a0 con 0x7f5e30004090 2024-03-23T14:36:56.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5e3000bb90 con 0x7f5e30004d00 2024-03-23T14:36:56.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5e3000bd50 con 0x7f5e440cf3b0 2024-03-23T14:36:56.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49b72640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e300047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e4a373640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e440cf3b0 0x7f5e30003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.550 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e4ab74640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e30004d00 0x7f5e30005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49b72640 1 -- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e440cf3b0 msgr2=0x7f5e30003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:56.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49b72640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e440cf3b0 0x7f5e30003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49b72640 1 -- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e30004d00 msgr2=0x7f5e30005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:56.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49b72640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e30004d00 0x7f5e30005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.543+0000 7f5e49b72640 1 -- 172.21.15.161:0/2473013083 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e30003520 con 0x7f5e30004090 2024-03-23T14:36:56.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.544+0000 7f5e49b72640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e300047c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5e34007fd0 tx=0x7f5e300057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8dce9c34b7aa61ad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:56.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.544+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e34005ad0 con 0x7f5e30004090 2024-03-23T14:36:56.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.544+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e34005c50 con 0x7f5e30004090 2024-03-23T14:36:56.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.544+0000 7f5e49371640 1 -- 172.21.15.161:0/2473013083 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e3402c830 con 0x7f5e30004090 2024-03-23T14:36:56.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.545+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/2473013083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 msgr2=0x7f5e300047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.545+0000 7f5e4c5d7640 1 --2- 172.21.15.161:0/2473013083 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e300047c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f5e34007fd0 tx=0x7f5e300057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.545+0000 7f5e4ab74640 1 -- 172.21.15.161:0/2473013083 reap_dead start 2024-03-23T14:36:56.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.545+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/2473013083 shutdown_connections 2024-03-23T14:36:56.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.545+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/2473013083 >> 172.21.15.161:0/2473013083 conn(0x7f5e4406b080 msgr2=0x7f5e4406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:56.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.545+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/2473013083 shutdown_connections 2024-03-23T14:36:56.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.545+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/2473013083 wait complete. 2024-03-23T14:36:56.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.546+0000 7f5e4c5d7640 1 Processor -- start 2024-03-23T14:36:56.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.546+0000 7f5e4c5d7640 1 -- start start 2024-03-23T14:36:56.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4c5d7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e4413d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4c5d7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e441e4070 0x7f5e441e6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4c5d7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e441e7100 0x7f5e441e3390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4c5d7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5e44159030 con 0x7f5e30004090 2024-03-23T14:36:56.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4c5d7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5e441e7c20 con 0x7f5e441e4070 2024-03-23T14:36:56.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4c5d7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5e441e7660 con 0x7f5e441e7100 2024-03-23T14:36:56.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e4413d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e4413d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45062/0 (socket says 172.21.15.161:45062) 2024-03-23T14:36:56.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 -- 172.21.15.161:0/1454853356 learned_addr learned my addr 172.21.15.161:0/1454853356 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:56.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e49b72640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e441e4070 0x7f5e441e6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4ab74640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e441e7100 0x7f5e441e3390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 -- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e441e7100 msgr2=0x7f5e441e3390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e441e7100 0x7f5e441e3390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 -- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e441e4070 msgr2=0x7f5e441e6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:56.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e441e4070 0x7f5e441e6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:56.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 -- 172.21.15.161:0/1454853356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e441e7940 con 0x7f5e30004090 2024-03-23T14:36:56.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e49b72640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e441e4070 0x7f5e441e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:56.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4a373640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e4413d4e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000fae0 tx=0x7f5e400021e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:56.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e3affd640 1 -- 172.21.15.161:0/1454853356 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40003b80 con 0x7f5e30004090 2024-03-23T14:36:56.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e3affd640 1 -- 172.21.15.161:0/1454853356 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e40003ce0 con 0x7f5e30004090 2024-03-23T14:36:56.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e441e1250 con 0x7f5e30004090 2024-03-23T14:36:56.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e3affd640 1 -- 172.21.15.161:0/1454853356 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4000daf0 con 0x7f5e30004090 2024-03-23T14:36:56.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.547+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e4413da20 con 0x7f5e30004090 2024-03-23T14:36:56.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.548+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5e441e10b0 con 0x7f5e30004090 2024-03-23T14:36:56.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.549+0000 7f5e3affd640 1 -- 172.21.15.161:0/1454853356 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5e4000dc50 con 0x7f5e30004090 2024-03-23T14:36:56.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.549+0000 7f5e3affd640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e2806b810 0x7f5e2806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:56.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.549+0000 7f5e3affd640 1 -- 172.21.15.161:0/1454853356 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(44..44 src has 1..44) ==== 4857+0+0 (secure 0 0 0) 0x7f5e4008a240 con 0x7f5e30004090 2024-03-23T14:36:56.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.549+0000 7f5e49b72640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e2806b810 0x7f5e2806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:56.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.550+0000 7f5e49b72640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e2806b810 0x7f5e2806dcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5e34034000 tx=0x7f5e34007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:56.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.554+0000 7f5e3affd640 1 -- 172.21.15.161:0/1454853356 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5e441e10b0 con 0x7f5e30004090 2024-03-23T14:36:56.726 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:56.729+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f5e441e10b0 con 0x7f5e30004090 2024-03-23T14:36:57.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.144+0000 7f5e3affd640 1 -- 172.21.15.161:0/1454853356 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v45) ==== 138+0+0 (secure 0 0 0) 0x7f5e441e10b0 con 0x7f5e30004090 2024-03-23T14:36:57.140 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' created 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e2806b810 msgr2=0x7f5e2806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e2806b810 0x7f5e2806dcd0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5e34034000 tx=0x7f5e34007000 comp rx=0 tx=0).stop 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 msgr2=0x7f5e4413d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e4413d4e0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000fae0 tx=0x7f5e400021e0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 shutdown_connections 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5e2806b810 0x7f5e2806dcd0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5e441e7100 0x7f5e441e3390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5e441e4070 0x7f5e441e6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 --2- 172.21.15.161:0/1454853356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5e30004090 0x7f5e4413d4e0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 >> 172.21.15.161:0/1454853356 conn(0x7f5e4406b080 msgr2=0x7f5e44145070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 shutdown_connections 2024-03-23T14:36:57.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.146+0000 7f5e4c5d7640 1 -- 172.21.15.161:0/1454853356 wait complete. 2024-03-23T14:36:57.156 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-23T14:36:57.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.273+0000 7f22a9864640 1 Processor -- start 2024-03-23T14:36:57.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.273+0000 7f22a9864640 1 -- start start 2024-03-23T14:36:57.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.274+0000 7f22a9864640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a415e440 0x7f22a40d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:57.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.274+0000 7f22a9864640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a40b0ac0 con 0x7f22a40d2bf0 2024-03-23T14:36:57.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.274+0000 7f22a9864640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22a40b4220 con 0x7f22a40d1d50 2024-03-23T14:36:57.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.274+0000 7f22a9864640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f22a40b3350 con 0x7f22a415e440 2024-03-23T14:36:57.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.274+0000 7f22a37fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f22a40d2bf0 0x7f22a40b46b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48512/0 (socket says 172.21.15.161:48512) 2024-03-23T14:36:57.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.274+0000 7f22a37fe640 1 -- 172.21.15.161:0/4004509177 learned_addr learned my addr 172.21.15.161:0/4004509177 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:57.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.274+0000 7f22a2ffd640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a415e440 0x7f22a40d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:57.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.274+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1782651858 0 0) 0x7f22a40b0ac0 con 0x7f22a40d2bf0 2024-03-23T14:36:57.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f228c003520 con 0x7f22a40d2bf0 2024-03-23T14:36:57.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 968199743 0 0) 0x7f228c003520 con 0x7f22a40d2bf0 2024-03-23T14:36:57.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a415e440 msgr2=0x7f22a40d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a415e440 0x7f22a40d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 >> v1:172.21.15.183:6789/0 conn(0x7f22a40d1d50 legacy=0x7f22a40d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a40b3610 con 0x7f22a40d2bf0 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f229c001670 con 0x7f22a40d2bf0 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 >> v1:172.21.15.161:6789/0 conn(0x7f22a40d2bf0 legacy=0x7f22a40b46b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f228c003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f228c004350 0x7f228c004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a415e440 0x7f228c005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:57.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22a40b3350 con 0x7f228c0039a0 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f228c00b980 con 0x7f228c004350 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f228c00bb40 con 0x7f22a415e440 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a2ffd640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f228c003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a37fe640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a415e440 0x7f228c005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a27fc640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f228c004350 0x7f228c004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a2ffd640 1 -- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a415e440 msgr2=0x7f228c005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a2ffd640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a415e440 0x7f228c005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a2ffd640 1 -- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f228c004350 msgr2=0x7f228c004820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a2ffd640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f228c004350 0x7f228c004820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a2ffd640 1 -- 172.21.15.161:0/4004509177 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f228c003520 con 0x7f228c0039a0 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.275+0000 7f22a27fc640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f228c004350 0x7f228c004820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.276+0000 7f22a2ffd640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f228c003d80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2294009fc0 tx=0x7f228c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fef8609ecd4915c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:57.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.276+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f229402cc20 con 0x7f228c0039a0 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.276+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229400f040 con 0x7f228c0039a0 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.276+0000 7f22a1ffb640 1 -- 172.21.15.161:0/4004509177 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294034440 con 0x7f228c0039a0 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.277+0000 7f22a9864640 1 -- 172.21.15.161:0/4004509177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 msgr2=0x7f228c003d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.277+0000 7f22a9864640 1 --2- 172.21.15.161:0/4004509177 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f228c003d80 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f2294009fc0 tx=0x7f228c0057f0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.277+0000 7f22a37fe640 1 -- 172.21.15.161:0/4004509177 reap_dead start 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.277+0000 7f22a9864640 1 -- 172.21.15.161:0/4004509177 shutdown_connections 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.277+0000 7f22a9864640 1 -- 172.21.15.161:0/4004509177 >> 172.21.15.161:0/4004509177 conn(0x7f22a40d3f00 msgr2=0x7f22a40d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.277+0000 7f22a9864640 1 -- 172.21.15.161:0/4004509177 shutdown_connections 2024-03-23T14:36:57.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.277+0000 7f22a9864640 1 -- 172.21.15.161:0/4004509177 wait complete. 2024-03-23T14:36:57.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.278+0000 7f22a9864640 1 Processor -- start 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.278+0000 7f22a9864640 1 -- start start 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a9864640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f22a415a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a9864640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a40a1720 0x7f22a4153420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a9864640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22a40b45b0 0x7f22a415eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a9864640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22a4153960 con 0x7f228c0039a0 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a9864640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f22a409f030 con 0x7f22a40b45b0 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a9864640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f22a409f310 con 0x7f22a40a1720 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f22a415a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f22a415a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45084/0 (socket says 172.21.15.161:45084) 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 -- 172.21.15.161:0/4075131663 learned_addr learned my addr 172.21.15.161:0/4075131663 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a37fe640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22a40b45b0 0x7f22a415eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a27fc640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a40a1720 0x7f22a4153420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:57.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 -- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a40a1720 msgr2=0x7f22a4153420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a40a1720 0x7f22a4153420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 -- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22a40b45b0 msgr2=0x7f22a415eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22a40b45b0 0x7f22a415eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 -- 172.21.15.161:0/4075131663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22a409f5f0 con 0x7f228c0039a0 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a37fe640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22a40b45b0 0x7f22a415eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a27fc640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a40a1720 0x7f22a4153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.279+0000 7f22a2ffd640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f22a415a850 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f229402c690 tx=0x7f2294009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.280+0000 7f228bfff640 1 -- 172.21.15.161:0/4075131663 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294034ea0 con 0x7f228c0039a0 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.280+0000 7f228bfff640 1 -- 172.21.15.161:0/4075131663 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f229402ce60 con 0x7f228c0039a0 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.280+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22a409f8d0 con 0x7f228c0039a0 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.280+0000 7f228bfff640 1 -- 172.21.15.161:0/4075131663 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294003bb0 con 0x7f228c0039a0 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.280+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22a40b4a60 con 0x7f228c0039a0 2024-03-23T14:36:57.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.281+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22a409fdf0 con 0x7f228c0039a0 2024-03-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.281+0000 7f228bfff640 1 -- 172.21.15.161:0/4075131663 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2294040450 con 0x7f228c0039a0 2024-03-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.281+0000 7f228bfff640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f229006b810 0x7f229006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.282+0000 7f228bfff640 1 -- 172.21.15.161:0/4075131663 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(45..45 src has 1..45) ==== 5227+0+0 (secure 0 0 0) 0x7f22940ac860 con 0x7f228c0039a0 2024-03-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.282+0000 7f22a27fc640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f229006b810 0x7f229006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.282+0000 7f22a27fc640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f229006b810 0x7f229006dcd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f22a40a1b00 tx=0x7f2298009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:57.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.286+0000 7f228bfff640 1 -- 172.21.15.161:0/4075131663 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f22a409fdf0 con 0x7f228c0039a0 2024-03-23T14:36:57.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:57.460+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f22a409fdf0 con 0x7f228c0039a0 2024-03-23T14:36:58.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.146+0000 7f228bfff640 1 -- 172.21.15.161:0/4075131663 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v46) ==== 170+0+0 (secure 0 0 0) 0x7f22a409fdf0 con 0x7f228c0039a0 2024-03-23T14:36:58.142 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' created 2024-03-23T14:36:58.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f229006b810 msgr2=0x7f229006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:58.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f229006b810 0x7f229006dcd0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f22a40a1b00 tx=0x7f2298009040 comp rx=0 tx=0).stop 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 msgr2=0x7f22a415a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f22a415a850 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f229402c690 tx=0x7f2294009750 comp rx=0 tx=0).stop 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 shutdown_connections 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f229006b810 0x7f229006dcd0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f22a40b45b0 0x7f22a415eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22a40a1720 0x7f22a4153420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 --2- 172.21.15.161:0/4075131663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f228c0039a0 0x7f22a415a850 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 >> 172.21.15.161:0/4075131663 conn(0x7f22a40d3f00 msgr2=0x7f22a40d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 shutdown_connections 2024-03-23T14:36:58.147 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.150+0000 7f22a9864640 1 -- 172.21.15.161:0/4075131663 wait complete. 2024-03-23T14:36:58.169 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-23T14:36:58.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.280+0000 7f4144119640 1 Processor -- start 2024-03-23T14:36:58.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.280+0000 7f4144119640 1 -- start start 2024-03-23T14:36:58.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4144119640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f413c06e950 0x7f413c15b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:58.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4144119640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c15ba00 con 0x7f413c1549c0 2024-03-23T14:36:58.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4144119640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f413c071330 con 0x7f413c15b5b0 2024-03-23T14:36:58.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4144119640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f413c0b2a10 con 0x7f413c06e950 2024-03-23T14:36:58.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4141eb5640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f413c1549c0 0x7f413c070c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48520/0 (socket says 172.21.15.161:48520) 2024-03-23T14:36:58.277 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4141eb5640 1 -- 172.21.15.161:0/948722091 learned_addr learned my addr 172.21.15.161:0/948722091 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f41416b4640 1 --2- 172.21.15.161:0/948722091 >> v2:172.21.15.183:3300/0 conn(0x7f413c06e950 0x7f413c15b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4140eb3640 1 -- 172.21.15.161:0/948722091 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1595620426 0 0) 0x7f413c15ba00 con 0x7f413c1549c0 2024-03-23T14:36:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4140eb3640 1 -- 172.21.15.161:0/948722091 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4128004630 con 0x7f413c1549c0 2024-03-23T14:36:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f41416b4640 1 -- 172.21.15.161:0/948722091 >> v1:172.21.15.183:6789/0 conn(0x7f413c15b5b0 legacy=0x7f413c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f4140eb3640 1 -- 172.21.15.161:0/948722091 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 786457889 0 0) 0x7f413c071330 con 0x7f413c15b5b0 2024-03-23T14:36:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f41416b4640 1 -- 172.21.15.161:0/948722091 >> v1:172.21.15.161:6789/0 conn(0x7f413c1549c0 legacy=0x7f413c070c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:58.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.281+0000 7f41416b4640 1 -- 172.21.15.161:0/948722091 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f413c06edd0 con 0x7f413c06e950 2024-03-23T14:36:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.282+0000 7f41416b4640 1 --2- 172.21.15.161:0/948722091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c06e950 0x7f413c15b070 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f413c0b2b50 tx=0x7f412c02db70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de70ea21147646d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.282+0000 7f4140eb3640 1 -- 172.21.15.161:0/948722091 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c02e920 con 0x7f413c06e950 2024-03-23T14:36:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.282+0000 7f4140eb3640 1 -- 172.21.15.161:0/948722091 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f412c02ea80 con 0x7f413c06e950 2024-03-23T14:36:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.282+0000 7f4140eb3640 1 -- 172.21.15.161:0/948722091 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c02edf0 con 0x7f413c06e950 2024-03-23T14:36:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.283+0000 7f4144119640 1 -- 172.21.15.161:0/948722091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c06e950 msgr2=0x7f413c15b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.283+0000 7f4144119640 1 --2- 172.21.15.161:0/948722091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c06e950 0x7f413c15b070 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f413c0b2b50 tx=0x7f412c02db70 comp rx=0 tx=0).stop 2024-03-23T14:36:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.283+0000 7f4144119640 1 -- 172.21.15.161:0/948722091 shutdown_connections 2024-03-23T14:36:58.280 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.283+0000 7f4144119640 1 --2- 172.21.15.161:0/948722091 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c06e950 0x7f413c15b070 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.283+0000 7f4144119640 1 -- 172.21.15.161:0/948722091 >> 172.21.15.161:0/948722091 conn(0x7f413c06b4a0 msgr2=0x7f413c06b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.283+0000 7f4144119640 1 -- 172.21.15.161:0/948722091 shutdown_connections 2024-03-23T14:36:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.283+0000 7f4144119640 1 -- 172.21.15.161:0/948722091 wait complete. 2024-03-23T14:36:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.284+0000 7f4144119640 1 Processor -- start 2024-03-23T14:36:58.281 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.284+0000 7f4144119640 1 -- start start 2024-03-23T14:36:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4144119640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f413c06e950 0x7f413c14ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4144119640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f413c1549c0 0x7f413c0afb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4144119640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c15b5b0 0x7f413c15cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4144119640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f413c0b0f80 con 0x7f413c1549c0 2024-03-23T14:36:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4144119640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f413c0ce5b0 con 0x7f413c15b5b0 2024-03-23T14:36:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4144119640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f413c0b0080 con 0x7f413c06e950 2024-03-23T14:36:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f413c1549c0 0x7f413c0afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:58.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f413c1549c0 0x7f413c0afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45096/0 (socket says 172.21.15.161:45096) 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 -- 172.21.15.161:0/3147296257 learned_addr learned my addr 172.21.15.161:0/3147296257 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4141eb5640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f413c06e950 0x7f413c14ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41426b6640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c15b5b0 0x7f413c15cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 -- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f413c06e950 msgr2=0x7f413c14ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f413c06e950 0x7f413c14ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 -- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c15b5b0 msgr2=0x7f413c15cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c15b5b0 0x7f413c15cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 -- 172.21.15.161:0/3147296257 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f412c009400 con 0x7f413c1549c0 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41416b4640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f413c1549c0 0x7f413c0afb40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f412c000c00 tx=0x7f412c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41327fc640 1 -- 172.21.15.161:0/3147296257 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c041620 con 0x7f413c1549c0 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41327fc640 1 -- 172.21.15.161:0/3147296257 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f412c041780 con 0x7f413c1549c0 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f413c0b0300 con 0x7f413c1549c0 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f41327fc640 1 -- 172.21.15.161:0/3147296257 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f412c041b10 con 0x7f413c1549c0 2024-03-23T14:36:58.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.285+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f413c0b08c0 con 0x7f413c1549c0 2024-03-23T14:36:58.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.286+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f413c0b10c0 con 0x7f413c1549c0 2024-03-23T14:36:58.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.287+0000 7f41327fc640 1 -- 172.21.15.161:0/3147296257 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f412c03d040 con 0x7f413c1549c0 2024-03-23T14:36:58.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.287+0000 7f41327fc640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f412006b810 0x7f412006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:58.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.287+0000 7f41327fc640 1 -- 172.21.15.161:0/3147296257 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(46..46 src has 1..46) ==== 5609+0+0 (secure 0 0 0) 0x7f412c0ae840 con 0x7f413c1549c0 2024-03-23T14:36:58.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.288+0000 7f4141eb5640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f412006b810 0x7f412006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:58.285 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.288+0000 7f4141eb5640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f412006b810 0x7f412006dcd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4138005880 tx=0x7f4138009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:58.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.292+0000 7f41327fc640 1 -- 172.21.15.161:0/3147296257 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f413c0b10c0 con 0x7f413c1549c0 2024-03-23T14:36:58.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:58.501+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f413c0b10c0 con 0x7f413c1549c0 2024-03-23T14:36:59.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.168+0000 7f41327fc640 1 -- 172.21.15.161:0/3147296257 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 7 and data pool 8 v21) ==== 172+0+0 (secure 0 0 0) 0x7f413c0b10c0 con 0x7f413c1549c0 2024-03-23T14:36:59.166 INFO:teuthology.orchestra.run.smithi161.stderr:new fs with metadata pool 7 and data pool 8 2024-03-23T14:36:59.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f412006b810 msgr2=0x7f412006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f412006b810 0x7f412006dcd0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f4138005880 tx=0x7f4138009040 comp rx=0 tx=0).stop 2024-03-23T14:36:59.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f413c1549c0 msgr2=0x7f413c0afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f413c1549c0 0x7f413c0afb40 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f412c000c00 tx=0x7f412c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 shutdown_connections 2024-03-23T14:36:59.169 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f412006b810 0x7f412006dcd0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f413c15b5b0 0x7f413c15cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f413c1549c0 0x7f413c0afb40 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 --2- 172.21.15.161:0/3147296257 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f413c06e950 0x7f413c14ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 >> 172.21.15.161:0/3147296257 conn(0x7f413c06b4a0 msgr2=0x7f413c0b16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:59.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 shutdown_connections 2024-03-23T14:36:59.170 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.173+0000 7f4144119640 1 -- 172.21.15.161:0/3147296257 wait complete. 2024-03-23T14:36:59.192 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:59.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.299+0000 7f8644c2e640 1 Processor -- start 2024-03-23T14:36:59.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.299+0000 7f8644c2e640 1 -- start start 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f8644c2e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f864006eb30 0x7f86400b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f8644c2e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86400b3610 0x7f86400b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f8644c2e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8640070b90 con 0x7f86400b4260 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f8644c2e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f86400b3fe0 con 0x7f864006eb30 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f8644c2e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f864015ab90 con 0x7f86400b3610 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863dd9b640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f86400b4260 0x7f864015b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48536/0 (socket says 172.21.15.161:48536) 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863dd9b640 1 -- 172.21.15.161:0/624259330 learned_addr learned my addr 172.21.15.161:0/624259330 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863e59c640 1 --2- 172.21.15.161:0/624259330 >> v2:172.21.15.183:3300/0 conn(0x7f864006eb30 0x7f86400b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863ed9d640 1 --2- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86400b3610 0x7f86400b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863d59a640 1 -- 172.21.15.161:0/624259330 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468494909 0 0) 0x7f8640070b90 con 0x7f86400b4260 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863d59a640 1 -- 172.21.15.161:0/624259330 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8628003520 con 0x7f86400b4260 2024-03-23T14:36:59.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863e59c640 1 -- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86400b3610 msgr2=0x7f86400b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863e59c640 1 --2- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86400b3610 0x7f86400b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863e59c640 1 -- 172.21.15.161:0/624259330 >> v1:172.21.15.161:6789/0 conn(0x7f86400b4260 legacy=0x7f864015b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863e59c640 1 -- 172.21.15.161:0/624259330 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f864015b0d0 con 0x7f864006eb30 2024-03-23T14:36:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.300+0000 7f863ed9d640 1 --2- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86400b3610 0x7f86400b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:36:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.301+0000 7f863e59c640 1 --2- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864006eb30 0x7f86400b4870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f86380097a0 tx=0x7f863802b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3271c1c1cb08ff1f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.301+0000 7f863d59a640 1 -- 172.21.15.161:0/624259330 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f863802e070 con 0x7f864006eb30 2024-03-23T14:36:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.301+0000 7f863d59a640 1 -- 172.21.15.161:0/624259330 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86380315a0 con 0x7f864006eb30 2024-03-23T14:36:59.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.301+0000 7f863d59a640 1 -- 172.21.15.161:0/624259330 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8638031910 con 0x7f864006eb30 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.302+0000 7f8644c2e640 1 -- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864006eb30 msgr2=0x7f86400b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.302+0000 7f8644c2e640 1 --2- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864006eb30 0x7f86400b4870 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f86380097a0 tx=0x7f863802b840 comp rx=0 tx=0).stop 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.302+0000 7f8644c2e640 1 -- 172.21.15.161:0/624259330 shutdown_connections 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.302+0000 7f8644c2e640 1 --2- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f86400b3610 0x7f86400b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.302+0000 7f8644c2e640 1 --2- 172.21.15.161:0/624259330 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f864006eb30 0x7f86400b4870 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.302+0000 7f8644c2e640 1 -- 172.21.15.161:0/624259330 >> 172.21.15.161:0/624259330 conn(0x7f86400b17f0 msgr2=0x7f86400b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.302+0000 7f8644c2e640 1 -- 172.21.15.161:0/624259330 shutdown_connections 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.302+0000 7f8644c2e640 1 -- 172.21.15.161:0/624259330 wait complete. 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.303+0000 7f8644c2e640 1 Processor -- start 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.303+0000 7f8644c2e640 1 -- start start 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f8644c2e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864006eb30 0x7f8640157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f8644c2e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86400b3610 0x7f8640152180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f8644c2e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86400b4260 0x7f86401526c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f8644c2e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8640158340 con 0x7f86400b3610 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f8644c2e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8640151cf0 con 0x7f86400b4260 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f8644c2e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f86401567d0 con 0x7f864006eb30 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86400b3610 0x7f8640152180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86400b3610 0x7f8640152180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45106/0 (socket says 172.21.15.161:45106) 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 -- 172.21.15.161:0/1335417754 learned_addr learned my addr 172.21.15.161:0/1335417754 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863ed9d640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86400b4260 0x7f86401526c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863e59c640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864006eb30 0x7f8640157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 -- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864006eb30 msgr2=0x7f8640157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864006eb30 0x7f8640157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 -- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86400b4260 msgr2=0x7f86401526c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86400b4260 0x7f86401526c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 -- 172.21.15.161:0/1335417754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8630009400 con 0x7f86400b3610 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863e59c640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864006eb30 0x7f8640157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f863dd9b640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86400b3610 0x7f8640152180 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f863400ee00 tx=0x7f863400c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.304+0000 7f862effd640 1 -- 172.21.15.161:0/1335417754 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634011670 con 0x7f86400b3610 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.305+0000 7f862effd640 1 -- 172.21.15.161:0/1335417754 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f86340117d0 con 0x7f86400b3610 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.305+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8638009400 con 0x7f86400b3610 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.305+0000 7f862effd640 1 -- 172.21.15.161:0/1335417754 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8634011b60 con 0x7f86400b3610 2024-03-23T14:36:59.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.305+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86401516a0 con 0x7f86400b3610 2024-03-23T14:36:59.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.306+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f86400b20c0 con 0x7f86400b3610 2024-03-23T14:36:59.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.306+0000 7f862effd640 1 -- 172.21.15.161:0/1335417754 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8634011cc0 con 0x7f86400b3610 2024-03-23T14:36:59.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.306+0000 7f862effd640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f862006b810 0x7f862006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.306+0000 7f862effd640 1 -- 172.21.15.161:0/1335417754 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5709+0+0 (secure 0 0 0) 0x7f863408d3e0 con 0x7f86400b3610 2024-03-23T14:36:59.304 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.306+0000 7f863e59c640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f862006b810 0x7f862006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.307+0000 7f863e59c640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f862006b810 0x7f862006dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8638036000 tx=0x7f8638008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.311+0000 7f862effd640 1 -- 172.21.15.161:0/1335417754 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8634057d80 con 0x7f86400b3610 2024-03-23T14:36:59.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.484+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f864004f550 con 0x7f86400b3610 2024-03-23T14:36:59.481 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.485+0000 7f862effd640 1 -- 172.21.15.161:0/1335417754 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+16718 (secure 0 0 0) 0x7f863405b440 con 0x7f86400b3610 2024-03-23T14:36:59.481 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:59.481 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":47,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:58.502885+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f862006b810 msgr2=0x7f862006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f862006b810 0x7f862006dcd0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f8638036000 tx=0x7f8638008030 comp rx=0 tx=0).stop 2024-03-23T14:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86400b3610 msgr2=0x7f8640152180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86400b3610 0x7f8640152180 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f863400ee00 tx=0x7f863400c640 comp rx=0 tx=0).stop 2024-03-23T14:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 shutdown_connections 2024-03-23T14:36:59.483 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f862006b810 0x7f862006dcd0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f86400b4260 0x7f86401526c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f86400b3610 0x7f8640152180 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 --2- 172.21.15.161:0/1335417754 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f864006eb30 0x7f8640157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 >> 172.21.15.161:0/1335417754 conn(0x7f86400b17f0 msgr2=0x7f864015cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 shutdown_connections 2024-03-23T14:36:59.484 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.487+0000 7f8644c2e640 1 -- 172.21.15.161:0/1335417754 wait complete. 2024-03-23T14:36:59.494 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:36:59.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.598+0000 7faa288c0640 1 Processor -- start 2024-03-23T14:36:59.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.598+0000 7faa288c0640 1 -- start start 2024-03-23T14:36:59.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.599+0000 7faa288c0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faa2415e400 0x7faa240cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.599+0000 7faa288c0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7faa240cf9d0 0x7faa240d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.599+0000 7faa288c0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 0x7faa240d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.599+0000 7faa288c0640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7faa240d1050 con 0x7faa2415e400 2024-03-23T14:36:59.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.599+0000 7faa288c0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7faa240d0460 con 0x7faa240cf9d0 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.599+0000 7faa288c0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7faa240d06b0 con 0x7faa240d1470 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.599+0000 7faa2259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faa2415e400 0x7faa240cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa2259c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7faa2415e400 0x7faa240cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45118/0 (socket says 172.21.15.161:45118) 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa2259c640 1 -- 172.21.15.161:0/4254111112 learned_addr learned my addr 172.21.15.161:0/4254111112 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa21d9b640 1 --2- 172.21.15.161:0/4254111112 >> v2:172.21.15.183:3300/0 conn(0x7faa240cf9d0 0x7faa240d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa22d9d640 1 --2- 172.21.15.161:0/4254111112 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 0x7faa240d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa2259c640 1 -- 172.21.15.161:0/4254111112 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 msgr2=0x7faa240d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa2259c640 1 --2- 172.21.15.161:0/4254111112 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 0x7faa240d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa2259c640 1 -- 172.21.15.161:0/4254111112 >> v2:172.21.15.183:3300/0 conn(0x7faa240cf9d0 msgr2=0x7faa240d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa2259c640 1 --2- 172.21.15.161:0/4254111112 >> v2:172.21.15.183:3300/0 conn(0x7faa240cf9d0 0x7faa240d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.600+0000 7faa2259c640 1 -- 172.21.15.161:0/4254111112 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa240d0990 con 0x7faa2415e400 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa2259c640 1 --2- 172.21.15.161:0/4254111112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa240cf490 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faa240b6a20 tx=0x7faa1c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a32a1bc9366697a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa2159a640 1 -- 172.21.15.161:0/4254111112 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c02ca70 con 0x7faa2415e400 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa2159a640 1 -- 172.21.15.161:0/4254111112 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1c02cbd0 con 0x7faa2415e400 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa2159a640 1 -- 172.21.15.161:0/4254111112 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1c0343f0 con 0x7faa2415e400 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 -- 172.21.15.161:0/4254111112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 msgr2=0x7faa240cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 --2- 172.21.15.161:0/4254111112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa240cf490 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7faa240b6a20 tx=0x7faa1c02bed0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 -- 172.21.15.161:0/4254111112 shutdown_connections 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 --2- 172.21.15.161:0/4254111112 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 0x7faa240d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 --2- 172.21.15.161:0/4254111112 >> v2:172.21.15.183:3300/0 conn(0x7faa240cf9d0 0x7faa240d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 --2- 172.21.15.161:0/4254111112 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa240cf490 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 -- 172.21.15.161:0/4254111112 >> 172.21.15.161:0/4254111112 conn(0x7faa2409d190 msgr2=0x7faa240d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 -- 172.21.15.161:0/4254111112 shutdown_connections 2024-03-23T14:36:59.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.601+0000 7faa288c0640 1 -- 172.21.15.161:0/4254111112 wait complete. 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.603+0000 7faa288c0640 1 Processor -- start 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.603+0000 7faa288c0640 1 -- start start 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.603+0000 7faa288c0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa240cf9d0 0x7faa2415b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.603+0000 7faa288c0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 0x7faa24161000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.603+0000 7faa288c0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa241f3d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.603+0000 7faa288c0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7faa240b9af0 con 0x7faa2415e400 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.603+0000 7faa288c0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7faa2415b8a0 con 0x7faa240cf9d0 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.603+0000 7faa288c0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7faa240b8ee0 con 0x7faa240d1470 2024-03-23T14:36:59.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa241f3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa241f3d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45122/0 (socket says 172.21.15.161:45122) 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 -- 172.21.15.161:0/2511299850 learned_addr learned my addr 172.21.15.161:0/2511299850 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa2259c640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa240cf9d0 0x7faa2415b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa21d9b640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 0x7faa24161000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 -- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 msgr2=0x7faa24161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 0x7faa24161000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 -- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa240cf9d0 msgr2=0x7faa2415b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa240cf9d0 0x7faa2415b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 -- 172.21.15.161:0/2511299850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1c009b30 con 0x7faa2415e400 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa22d9d640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa241f3d40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7faa1400d800 tx=0x7faa1400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa1affd640 1 -- 172.21.15.161:0/2511299850 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1400b6c0 con 0x7faa2415e400 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa1affd640 1 -- 172.21.15.161:0/2511299850 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faa1400b840 con 0x7faa2415e400 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa240b91c0 con 0x7faa2415e400 2024-03-23T14:36:59.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa1affd640 1 -- 172.21.15.161:0/2511299850 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa1400bbf0 con 0x7faa2415e400 2024-03-23T14:36:59.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.604+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa240b9780 con 0x7faa2415e400 2024-03-23T14:36:59.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.605+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa240b9c60 con 0x7faa2415e400 2024-03-23T14:36:59.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.606+0000 7faa1affd640 1 -- 172.21.15.161:0/2511299850 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7faa1400bd50 con 0x7faa2415e400 2024-03-23T14:36:59.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.606+0000 7faa1affd640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faa0406b810 0x7faa0406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.606+0000 7faa1affd640 1 -- 172.21.15.161:0/2511299850 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5709+0+0 (secure 0 0 0) 0x7faa1408a730 con 0x7faa2415e400 2024-03-23T14:36:59.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.606+0000 7faa2259c640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faa0406b810 0x7faa0406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.607+0000 7faa2259c640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faa0406b810 0x7faa0406dcd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faa1c035000 tx=0x7faa1c030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.606 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.610+0000 7faa1affd640 1 -- 172.21.15.161:0/2511299850 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7faa14055150 con 0x7faa2415e400 2024-03-23T14:36:59.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.783+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7faa240a1e00 con 0x7faa2415e400 2024-03-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.784+0000 7faa1affd640 1 -- 172.21.15.161:0/2511299850 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) ==== 74+0+16718 (secure 0 0 0) 0x7faa14058810 con 0x7faa2415e400 2024-03-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:36:59.780 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":47,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:36:58.502885+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:36:59.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faa0406b810 msgr2=0x7faa0406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faa0406b810 0x7faa0406dcd0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7faa1c035000 tx=0x7faa1c030000 comp rx=0 tx=0).stop 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 msgr2=0x7faa241f3d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa241f3d40 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7faa1400d800 tx=0x7faa1400dcc0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 shutdown_connections 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7faa0406b810 0x7faa0406dcd0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7faa2415e400 0x7faa241f3d40 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7faa240d1470 0x7faa24161000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 --2- 172.21.15.161:0/2511299850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7faa240cf9d0 0x7faa2415b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 >> 172.21.15.161:0/2511299850 conn(0x7faa2409d190 msgr2=0x7faa2406bd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 shutdown_connections 2024-03-23T14:36:59.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.786+0000 7faa288c0640 1 -- 172.21.15.161:0/2511299850 wait complete. 2024-03-23T14:36:59.793 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-23T14:36:59.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.901+0000 7f9e061d0640 1 Processor -- start 2024-03-23T14:36:59.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.901+0000 7f9e061d0640 1 -- start start 2024-03-23T14:36:59.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9e061d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e0006e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9e061d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e000b2c20 con 0x7f9e00070c20 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9e061d0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e000b2f00 con 0x7f9e0015b780 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9e061d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9e0006ee90 con 0x7f9e001549c0 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dfeffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9e00070c20 0x7f9e0015b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:48550/0 (socket says 172.21.15.161:48550) 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dfeffd640 1 -- 172.21.15.161:0/1894850226 learned_addr learned my addr 172.21.15.161:0/1894850226 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dff7fe640 1 --2- 172.21.15.161:0/1894850226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e0006e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dfe7fc640 1 -- 172.21.15.161:0/1894850226 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4046188089 0 0) 0x7f9e000b2c20 con 0x7f9e00070c20 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dfe7fc640 1 -- 172.21.15.161:0/1894850226 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de8003520 con 0x7f9e00070c20 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dfe7fc640 1 -- 172.21.15.161:0/1894850226 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2972088154 0 0) 0x7f9e000b2f00 con 0x7f9e0015b780 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dfe7fc640 1 -- 172.21.15.161:0/1894850226 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9e000b2c20 con 0x7f9e0015b780 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dff7fe640 1 -- 172.21.15.161:0/1894850226 >> v1:172.21.15.183:6789/0 conn(0x7f9e0015b780 legacy=0x7f9e0006d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dff7fe640 1 -- 172.21.15.161:0/1894850226 >> v1:172.21.15.161:6789/0 conn(0x7f9e00070c20 legacy=0x7f9e0015b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.902+0000 7f9dff7fe640 1 -- 172.21.15.161:0/1894850226 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e000b31e0 con 0x7f9e001549c0 2024-03-23T14:36:59.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.903+0000 7f9dff7fe640 1 --2- 172.21.15.161:0/1894850226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e0006e950 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9df00097d0 tx=0x7f9df002b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f2e8b0fac9d764db server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.903+0000 7f9dfe7fc640 1 -- 172.21.15.161:0/1894850226 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0031890 con 0x7f9e001549c0 2024-03-23T14:36:59.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.903+0000 7f9dfe7fc640 1 -- 172.21.15.161:0/1894850226 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df00319f0 con 0x7f9e001549c0 2024-03-23T14:36:59.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.903+0000 7f9dfe7fc640 1 -- 172.21.15.161:0/1894850226 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0031d60 con 0x7f9e001549c0 2024-03-23T14:36:59.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.904+0000 7f9e061d0640 1 -- 172.21.15.161:0/1894850226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 msgr2=0x7f9e0006e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.904+0000 7f9e061d0640 1 --2- 172.21.15.161:0/1894850226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e0006e950 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9df00097d0 tx=0x7f9df002b930 comp rx=0 tx=0).stop 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.904+0000 7f9e061d0640 1 -- 172.21.15.161:0/1894850226 shutdown_connections 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.904+0000 7f9e061d0640 1 --2- 172.21.15.161:0/1894850226 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e0006e950 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.904+0000 7f9e061d0640 1 -- 172.21.15.161:0/1894850226 >> 172.21.15.161:0/1894850226 conn(0x7f9e0006b4a0 msgr2=0x7f9e0006b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.904+0000 7f9e061d0640 1 -- 172.21.15.161:0/1894850226 shutdown_connections 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.904+0000 7f9e061d0640 1 -- 172.21.15.161:0/1894850226 wait complete. 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.905+0000 7f9e061d0640 1 Processor -- start 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.905+0000 7f9e061d0640 1 -- start start 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9e061d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e00070c20 0x7f9e000b1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9e061d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e000ce930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9e061d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0015b780 0x7f9e000cee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9e061d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9e001463d0 con 0x7f9e0015b780 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9e061d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000b1700 con 0x7f9e00070c20 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9e061d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9e000b0b30 con 0x7f9e001549c0 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0015b780 0x7f9e000cee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0015b780 0x7f9e000cee70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45132/0 (socket says 172.21.15.161:45132) 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 -- 172.21.15.161:0/16058186 learned_addr learned my addr 172.21.15.161:0/16058186 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dff7fe640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e00070c20 0x7f9e000b1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfeffd640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e000ce930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 -- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 msgr2=0x7f9e000ce930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e000ce930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 -- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e00070c20 msgr2=0x7f9e000b1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e00070c20 0x7f9e000b1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 -- 172.21.15.161:0/16058186 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9df0009400 con 0x7f9e0015b780 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfeffd640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e000ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9dfffff640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0015b780 0x7f9e000cee70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9df800c6b0 tx=0x7f9df800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.906+0000 7f9de7fff640 1 -- 172.21.15.161:0/16058186 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8003b70 con 0x7f9e0015b780 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.907+0000 7f9de7fff640 1 -- 172.21.15.161:0/16058186 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df8003cd0 con 0x7f9e0015b780 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.907+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e000b0dd0 con 0x7f9e0015b780 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.907+0000 7f9de7fff640 1 -- 172.21.15.161:0/16058186 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df8005310 con 0x7f9e0015b780 2024-03-23T14:36:59.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.907+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e000b1390 con 0x7f9e0015b780 2024-03-23T14:36:59.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.908+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e000b1870 con 0x7f9e0015b780 2024-03-23T14:36:59.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.908+0000 7f9de7fff640 1 -- 172.21.15.161:0/16058186 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9df8005470 con 0x7f9e0015b780 2024-03-23T14:36:59.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.908+0000 7f9de7fff640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9dec06b810 0x7f9dec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:36:59.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.909+0000 7f9de7fff640 1 -- 172.21.15.161:0/16058186 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 5709+0+0 (secure 0 0 0) 0x7f9df8089f00 con 0x7f9e0015b780 2024-03-23T14:36:59.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.909+0000 7f9dff7fe640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9dec06b810 0x7f9dec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:36:59.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.909+0000 7f9dff7fe640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9dec06b810 0x7f9dec06dcd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9df0036000 tx=0x7f9df003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:36:59.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:36:59.913+0000 7f9de7fff640 1 -- 172.21.15.161:0/16058186 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9df8054920 con 0x7f9e0015b780 2024-03-23T14:37:00.117 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.121+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f9e00073570 con 0x7f9e0015b780 2024-03-23T14:37:00.163 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.168+0000 7f9de7fff640 1 -- 172.21.15.161:0/16058186 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v22) ==== 118+0+0 (secure 0 0 0) 0x7f9df8057fe0 con 0x7f9e0015b780 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9dec06b810 msgr2=0x7f9dec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9dec06b810 0x7f9dec06dcd0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f9df0036000 tx=0x7f9df003c040 comp rx=0 tx=0).stop 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0015b780 msgr2=0x7f9e000cee70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0015b780 0x7f9e000cee70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f9df800c6b0 tx=0x7f9df800cb70 comp rx=0 tx=0).stop 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 shutdown_connections 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9dec06b810 0x7f9dec06dcd0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9e0015b780 0x7f9e000cee70 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9e001549c0 0x7f9e000ce930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 --2- 172.21.15.161:0/16058186 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9e00070c20 0x7f9e000b1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 >> 172.21.15.161:0/16058186 conn(0x7f9e0006b4a0 msgr2=0x7f9e0015d610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:00.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 shutdown_connections 2024-03-23T14:37:00.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.171+0000 7f9e061d0640 1 -- 172.21.15.161:0/16058186 wait complete. 2024-03-23T14:37:00.179 DEBUG:tasks.cephfs.filesystem:fs_config: {} 2024-03-23T14:37:00.179 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-03-23T14:37:00.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.288+0000 7fd016a55640 1 Processor -- start 2024-03-23T14:37:00.284 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.288+0000 7fd016a55640 1 -- start start 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd016a55640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd010146d60 0x7fd0100ad8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd016a55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd010161c40 0x7fd0101620c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd016a55640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd010162700 con 0x7fd0100721f0 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd016a55640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fd0101629e0 con 0x7fd010146d60 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd016a55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd0100b43c0 con 0x7fd010161c40 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd010146d60 0x7fd0100ad8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fd010146d60 0x7fd0100ad8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45146/0 (socket says 172.21.15.161:45146) 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00ffff640 1 -- 172.21.15.161:0/1543643371 learned_addr learned my addr 172.21.15.161:0/1543643371 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00f7fe640 1 --2- 172.21.15.161:0/1543643371 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd010161c40 0x7fd0101620c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00ffff640 1 -- 172.21.15.161:0/1543643371 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd010161c40 msgr2=0x7fd0101620c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00ffff640 1 --2- 172.21.15.161:0/1543643371 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd010161c40 0x7fd0101620c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00ffff640 1 -- 172.21.15.161:0/1543643371 >> v1:172.21.15.183:6789/0 conn(0x7fd0100721f0 legacy=0x7fd0100b4bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:00.286 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00ffff640 1 -- 172.21.15.161:0/1543643371 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0100b46a0 con 0x7fd010146d60 2024-03-23T14:37:00.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.289+0000 7fd00f7fe640 1 --2- 172.21.15.161:0/1543643371 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd010161c40 0x7fd0101620c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:00.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.290+0000 7fd00ffff640 1 --2- 172.21.15.161:0/1543643371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010146d60 0x7fd0100ad8b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd000009870 tx=0x7fd00002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e31ae5e269930a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:00.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.290+0000 7fd00effd640 1 -- 172.21.15.161:0/1543643371 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000030840 con 0x7fd010146d60 2024-03-23T14:37:00.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.290+0000 7fd00effd640 1 -- 172.21.15.161:0/1543643371 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd0000309a0 con 0x7fd010146d60 2024-03-23T14:37:00.287 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.290+0000 7fd00effd640 1 -- 172.21.15.161:0/1543643371 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd000030d30 con 0x7fd010146d60 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.290+0000 7fd016a55640 1 -- 172.21.15.161:0/1543643371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010146d60 msgr2=0x7fd0100ad8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.290+0000 7fd016a55640 1 --2- 172.21.15.161:0/1543643371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010146d60 0x7fd0100ad8b0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7fd000009870 tx=0x7fd00002b980 comp rx=0 tx=0).stop 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.291+0000 7fd016a55640 1 -- 172.21.15.161:0/1543643371 shutdown_connections 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.291+0000 7fd016a55640 1 --2- 172.21.15.161:0/1543643371 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd010161c40 0x7fd0101620c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.291+0000 7fd016a55640 1 --2- 172.21.15.161:0/1543643371 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010146d60 0x7fd0100ad8b0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.291+0000 7fd016a55640 1 -- 172.21.15.161:0/1543643371 >> 172.21.15.161:0/1543643371 conn(0x7fd01006ce50 msgr2=0x7fd01006d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.291+0000 7fd016a55640 1 -- 172.21.15.161:0/1543643371 shutdown_connections 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.291+0000 7fd016a55640 1 -- 172.21.15.161:0/1543643371 wait complete. 2024-03-23T14:37:00.288 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.292+0000 7fd016a55640 1 Processor -- start 2024-03-23T14:37:00.289 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.293+0000 7fd016a55640 1 -- start start 2024-03-23T14:37:00.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd016a55640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0100721f0 0x7fd010140550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:00.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd016a55640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd010146d60 0x7fd01013b620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:00.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd016a55640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010161c40 0x7fd010139cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:00.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd016a55640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd01013bb60 con 0x7fd010161c40 2024-03-23T14:37:00.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd016a55640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd01013a580 con 0x7fd010146d60 2024-03-23T14:37:00.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd016a55640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd01013a860 con 0x7fd0100721f0 2024-03-23T14:37:00.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010161c40 0x7fd010139cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010161c40 0x7fd010139cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45156/0 (socket says 172.21.15.161:45156) 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 -- 172.21.15.161:0/2453777082 learned_addr learned my addr 172.21.15.161:0/2453777082 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 -- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0100721f0 msgr2=0x7fd010140550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd00ffff640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0100721f0 0x7fd010140550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd00f7fe640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd010146d60 0x7fd01013b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0100721f0 0x7fd010140550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 -- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd010146d60 msgr2=0x7fd01013b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd010146d60 0x7fd01013b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 -- 172.21.15.161:0/2453777082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0000094d0 con 0x7fd010161c40 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.294+0000 7fd014ff2640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010161c40 0x7fd010139cf0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd00800c7a0 tx=0x7fd00800cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.295+0000 7fd00cff9640 1 -- 172.21.15.161:0/2453777082 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd008003b70 con 0x7fd010161c40 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.295+0000 7fd00cff9640 1 -- 172.21.15.161:0/2453777082 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd008003cf0 con 0x7fd010161c40 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.295+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd01013ab40 con 0x7fd010161c40 2024-03-23T14:37:00.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.295+0000 7fd00cff9640 1 -- 172.21.15.161:0/2453777082 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd008003ea0 con 0x7fd010161c40 2024-03-23T14:37:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.295+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd01013b100 con 0x7fd010161c40 2024-03-23T14:37:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.296+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0101388f0 con 0x7fd010161c40 2024-03-23T14:37:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.296+0000 7fd00cff9640 1 -- 172.21.15.161:0/2453777082 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd00800aa90 con 0x7fd010161c40 2024-03-23T14:37:00.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.296+0000 7fd00cff9640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcffc06b810 0x7fcffc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:00.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.297+0000 7fd00cff9640 1 -- 172.21.15.161:0/2453777082 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fd008089fb0 con 0x7fd010161c40 2024-03-23T14:37:00.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.297+0000 7fd00ffff640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcffc06b810 0x7fcffc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:00.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.297+0000 7fd00ffff640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcffc06b810 0x7fcffc06dcd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd000036000 tx=0x7fd00003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:00.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.302+0000 7fd00cff9640 1 -- 172.21.15.161:0/2453777082 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd0080bde10 con 0x7fd010161c40 2024-03-23T14:37:00.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:00.599+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7fd0100afaf0 con 0x7fd010161c40 2024-03-23T14:37:01.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.176+0000 7fd00cff9640 1 -- 172.21.15.161:0/2453777082 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v23) ==== 122+0+0 (secure 0 0 0) 0x7fd008054a60 con 0x7fd010161c40 2024-03-23T14:37:01.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcffc06b810 msgr2=0x7fcffc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcffc06b810 0x7fcffc06dcd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fd000036000 tx=0x7fd00003b040 comp rx=0 tx=0).stop 2024-03-23T14:37:01.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010161c40 msgr2=0x7fd010139cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010161c40 0x7fd010139cf0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fd00800c7a0 tx=0x7fd00800cc60 comp rx=0 tx=0).stop 2024-03-23T14:37:01.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 shutdown_connections 2024-03-23T14:37:01.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcffc06b810 0x7fcffc06dcd0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd010161c40 0x7fd010139cf0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd010146d60 0x7fd01013b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.176 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 --2- 172.21.15.161:0/2453777082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd0100721f0 0x7fd010140550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 >> 172.21.15.161:0/2453777082 conn(0x7fd01006ce50 msgr2=0x7fd0100b4ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:01.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 shutdown_connections 2024-03-23T14:37:01.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.178+0000 7fd016a55640 1 -- 172.21.15.161:0/2453777082 wait complete. 2024-03-23T14:37:01.188 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-03-23T14:37:01.188 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:37:01.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.295+0000 7f03a929d640 1 Processor -- start 2024-03-23T14:37:01.291 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.295+0000 7f03a929d640 1 -- start start 2024-03-23T14:37:01.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a929d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a415e400 0x7f03a40cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a929d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f03a40d2b40 0x7f03a40b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.292 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a929d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f03a40d0460 con 0x7f03a40cf9d0 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a929d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f03a40d1050 con 0x7f03a40d2b40 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a929d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f03a40d06b0 con 0x7f03a415e400 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a37fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f03a40d2b40 0x7f03a40b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a2ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a415e400 0x7f03a40cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f039a7fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f03a40cf9d0 0x7f03a40d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:46424/0 (socket says 172.21.15.161:46424) 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a37fe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f03a40d2b40 0x7f03a40b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:44186/0 (socket says 172.21.15.161:44186) 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f03a2ffd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a415e400 0x7f03a40cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:47156/0 (socket says 172.21.15.161:47156) 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.296+0000 7f039a7fc640 1 -- 172.21.15.161:0/604089020 learned_addr learned my addr 172.21.15.161:0/604089020 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.297+0000 7f03a37fe640 1 -- 172.21.15.161:0/604089020 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a415e400 msgr2=0x7f03a40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.297+0000 7f03a37fe640 1 --2- 172.21.15.161:0/604089020 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a415e400 0x7f03a40cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.297+0000 7f03a37fe640 1 -- 172.21.15.161:0/604089020 >> v1:172.21.15.183:6789/0 conn(0x7f03a40cf9d0 legacy=0x7f03a40d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.293 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.297+0000 7f03a37fe640 1 -- 172.21.15.161:0/604089020 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03a40d0990 con 0x7f03a40d2b40 2024-03-23T14:37:01.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.297+0000 7f03a27fc640 1 -- 172.21.15.161:0/604089020 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4068819610 0 0) 0x7f03a40d0460 con 0x7f03a40cf9d0 2024-03-23T14:37:01.294 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.297+0000 7f03a2ffd640 1 --2- 172.21.15.161:0/604089020 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a415e400 0x7f03a40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:01.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.297+0000 7f03a37fe640 1 --2- 172.21.15.161:0/604089020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a40d2b40 0x7f03a40b0a80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f039c009d90 tx=0x7f039c02bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eef7ebda715ec3eb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.297+0000 7f03a27fc640 1 -- 172.21.15.161:0/604089020 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c02cc50 con 0x7f03a40d2b40 2024-03-23T14:37:01.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a27fc640 1 -- 172.21.15.161:0/604089020 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f039c02cdb0 con 0x7f03a40d2b40 2024-03-23T14:37:01.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a27fc640 1 -- 172.21.15.161:0/604089020 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f039c034660 con 0x7f03a40d2b40 2024-03-23T14:37:01.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a929d640 1 -- 172.21.15.161:0/604089020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a40d2b40 msgr2=0x7f03a40b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.295 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a929d640 1 --2- 172.21.15.161:0/604089020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a40d2b40 0x7f03a40b0a80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f039c009d90 tx=0x7f039c02bea0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a929d640 1 -- 172.21.15.161:0/604089020 shutdown_connections 2024-03-23T14:37:01.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a929d640 1 --2- 172.21.15.161:0/604089020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a40d2b40 0x7f03a40b0a80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a929d640 1 --2- 172.21.15.161:0/604089020 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a415e400 0x7f03a40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a929d640 1 -- 172.21.15.161:0/604089020 >> 172.21.15.161:0/604089020 conn(0x7f03a409d190 msgr2=0x7f03a40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:01.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a929d640 1 -- 172.21.15.161:0/604089020 shutdown_connections 2024-03-23T14:37:01.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.298+0000 7f03a929d640 1 -- 172.21.15.161:0/604089020 wait complete. 2024-03-23T14:37:01.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.299+0000 7f03a929d640 1 Processor -- start 2024-03-23T14:37:01.296 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.299+0000 7f03a929d640 1 -- start start 2024-03-23T14:37:01.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f03a929d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a40cf9d0 0x7f03a415cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f03a929d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03a40d2b40 0x7f03a40b6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f03a929d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a415e400 0x7f03a40b89e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f03a929d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f03a40b7370 con 0x7f03a40d2b40 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f03a929d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f03a40b7650 con 0x7f03a415e400 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f03a929d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f03a40b7930 con 0x7f03a40cf9d0 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f039a7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03a40d2b40 0x7f03a40b6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f039a7fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03a40d2b40 0x7f03a40b6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45042/0 (socket says 172.21.15.161:45042) 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f039a7fc640 1 -- 172.21.15.161:0/3846430919 learned_addr learned my addr 172.21.15.161:0/3846430919 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f03a37fe640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a415e400 0x7f03a40b89e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.300+0000 7f03a2ffd640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a40cf9d0 0x7f03a415cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039a7fc640 1 -- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a40cf9d0 msgr2=0x7f03a415cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039a7fc640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a40cf9d0 0x7f03a415cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039a7fc640 1 -- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a415e400 msgr2=0x7f03a40b89e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039a7fc640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a415e400 0x7f03a40b89e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039a7fc640 1 -- 172.21.15.161:0/3846430919 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0394009400 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039a7fc640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03a40d2b40 0x7f03a40b6a00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f038c00c4d0 tx=0x7f038c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039bfff640 1 -- 172.21.15.161:0/3846430919 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c0074e0 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039bfff640 1 -- 172.21.15.161:0/3846430919 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f038c007660 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f039bfff640 1 -- 172.21.15.161:0/3846430919 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f038c004a20 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f039c009a70 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.301+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03a40b7ef0 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.302+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03a40b8110 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.302+0000 7f039bfff640 1 -- 172.21.15.161:0/3846430919 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f038c004b80 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.303+0000 7f039bfff640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f038806b840 0x7f038806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.303+0000 7f039bfff640 1 -- 172.21.15.161:0/3846430919 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f038c0895b0 con 0x7f03a40d2b40 2024-03-23T14:37:01.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.303+0000 7f03a2ffd640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f038806b840 0x7f038806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.301 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.304+0000 7f03a2ffd640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f038806b840 0x7f038806dd00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0394009c30 tx=0x7f0394009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.311+0000 7f039bfff640 1 -- 172.21.15.161:0/3846430919 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f038c054110 con 0x7f03a40d2b40 2024-03-23T14:37:01.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.524+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f03a40a1e00 con 0x7f03a40d2b40 2024-03-23T14:37:01.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.525+0000 7f039bfff640 1 -- 172.21.15.161:0/3846430919 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5193 (secure 0 0 0) 0x7f038c0577d0 con 0x7f03a40d2b40 2024-03-23T14:37:01.521 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:01.521 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2496234294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":2496234294},{"type":"v1","addr":"172.21.15.183:6836","nonce":2496234294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5396,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2645821131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2645821131},{"type":"v1","addr":"172.21.15.183:6837","nonce":2645821131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/2183619206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2183619206},{"type":"v1","addr":"172.21.15.161:6834","nonce":2183619206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:58.503230+0000","modified":"2024-03-23T14:37:00.601296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/1671955402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":1671955402},{"type":"v1","addr":"172.21.15.161:6835","nonce":1671955402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5925,"qdb_cluster":[5925]},"id":3}]} 2024-03-23T14:37:01.521 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f038806b840 msgr2=0x7f038806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f038806b840 0x7f038806dd00 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0394009c30 tx=0x7f0394009190 comp rx=0 tx=0).stop 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03a40d2b40 msgr2=0x7f03a40b6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03a40d2b40 0x7f03a40b6a00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7f038c00c4d0 tx=0x7f038c00c990 comp rx=0 tx=0).stop 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 shutdown_connections 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f038806b840 0x7f038806dd00 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f03a415e400 0x7f03a40b89e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f03a40d2b40 0x7f03a40b6a00 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.523 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 --2- 172.21.15.161:0/3846430919 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f03a40cf9d0 0x7f03a415cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 >> 172.21.15.161:0/3846430919 conn(0x7f03a409d190 msgr2=0x7f03a40b19d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:01.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 shutdown_connections 2024-03-23T14:37:01.524 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.527+0000 7f03a929d640 1 -- 172.21.15.161:0/3846430919 wait complete. 2024-03-23T14:37:01.534 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:01.634 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.639+0000 7fc99afef640 1 Processor -- start 2024-03-23T14:37:01.635 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.639+0000 7fc99afef640 1 -- start start 2024-03-23T14:37:01.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.639+0000 7fc99afef640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc99415e400 0x7fc9940cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.639+0000 7fc99afef640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc9940cf9d0 0x7fc9940d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.639+0000 7fc99afef640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9940d1470 0x7fc9940d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.639+0000 7fc99afef640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc9940d1050 con 0x7fc9940d1470 2024-03-23T14:37:01.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.639+0000 7fc99afef640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc9940d0460 con 0x7fc9940cf9d0 2024-03-23T14:37:01.637 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.639+0000 7fc99afef640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc9940d06b0 con 0x7fc99415e400 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc99958c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9940d1470 0x7fc9940d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc99958c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc9940d1470 0x7fc9940d2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45052/0 (socket says 172.21.15.161:45052) 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc99958c640 1 -- 172.21.15.161:0/2682823933 learned_addr learned my addr 172.21.15.161:0/2682823933 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc993fff640 1 --2- 172.21.15.161:0/2682823933 >> v2:172.21.15.183:3300/0 conn(0x7fc9940cf9d0 0x7fc9940d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc998d8b640 1 --2- 172.21.15.161:0/2682823933 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc99415e400 0x7fc9940cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc99958c640 1 -- 172.21.15.161:0/2682823933 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc99415e400 msgr2=0x7fc9940cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc99958c640 1 --2- 172.21.15.161:0/2682823933 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc99415e400 0x7fc9940cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc99958c640 1 -- 172.21.15.161:0/2682823933 >> v2:172.21.15.183:3300/0 conn(0x7fc9940cf9d0 msgr2=0x7fc9940d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc99958c640 1 --2- 172.21.15.161:0/2682823933 >> v2:172.21.15.183:3300/0 conn(0x7fc9940cf9d0 0x7fc9940d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc99958c640 1 -- 172.21.15.161:0/2682823933 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9940d0990 con 0x7fc9940d1470 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.640+0000 7fc998d8b640 1 --2- 172.21.15.161:0/2682823933 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc99415e400 0x7fc9940cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99958c640 1 --2- 172.21.15.161:0/2682823933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9940d1470 0x7fc9940d2970 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc99408e350 tx=0x7fc98c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=629a5df5feb5b87f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.638 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc9937fe640 1 -- 172.21.15.161:0/2682823933 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c02ca70 con 0x7fc9940d1470 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc9937fe640 1 -- 172.21.15.161:0/2682823933 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc98c02cbd0 con 0x7fc9940d1470 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc9937fe640 1 -- 172.21.15.161:0/2682823933 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c0343f0 con 0x7fc9940d1470 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 -- 172.21.15.161:0/2682823933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9940d1470 msgr2=0x7fc9940d2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 --2- 172.21.15.161:0/2682823933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9940d1470 0x7fc9940d2970 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fc99408e350 tx=0x7fc98c02bed0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 -- 172.21.15.161:0/2682823933 shutdown_connections 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 --2- 172.21.15.161:0/2682823933 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc9940d1470 0x7fc9940d2970 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 --2- 172.21.15.161:0/2682823933 >> v2:172.21.15.183:3300/0 conn(0x7fc9940cf9d0 0x7fc9940d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 --2- 172.21.15.161:0/2682823933 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc99415e400 0x7fc9940cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 -- 172.21.15.161:0/2682823933 >> 172.21.15.161:0/2682823933 conn(0x7fc99409d190 msgr2=0x7fc9940d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 -- 172.21.15.161:0/2682823933 shutdown_connections 2024-03-23T14:37:01.639 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.641+0000 7fc99afef640 1 -- 172.21.15.161:0/2682823933 wait complete. 2024-03-23T14:37:01.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.642+0000 7fc99afef640 1 Processor -- start 2024-03-23T14:37:01.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.642+0000 7fc99afef640 1 -- start start 2024-03-23T14:37:01.642 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99afef640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9940cf9d0 0x7fc9940b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99afef640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9940d1470 0x7fc99415d130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99afef640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415e400 0x7fc9940b8e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99afef640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc9940c89e0 con 0x7fc99415e400 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99afef640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc9940b6aa0 con 0x7fc9940cf9d0 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99afef640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc9940b6d80 con 0x7fc9940d1470 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99958c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415e400 0x7fc9940b8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99958c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415e400 0x7fc9940b8e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45064/0 (socket says 172.21.15.161:45064) 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.643+0000 7fc99958c640 1 -- 172.21.15.161:0/2979034340 learned_addr learned my addr 172.21.15.161:0/2979034340 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc998d8b640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9940cf9d0 0x7fc9940b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc993fff640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9940d1470 0x7fc99415d130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc99958c640 1 -- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9940d1470 msgr2=0x7fc99415d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc99958c640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9940d1470 0x7fc99415d130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc99958c640 1 -- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9940cf9d0 msgr2=0x7fc9940b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.643 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc99958c640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9940cf9d0 0x7fc9940b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc99958c640 1 -- 172.21.15.161:0/2979034340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98c009b30 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc99958c640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415e400 0x7fc9940b8e20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc98c036040 tx=0x7fc98c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc9917fa640 1 -- 172.21.15.161:0/2979034340 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c009510 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc9917fa640 1 -- 172.21.15.161:0/2979034340 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc98c03e450 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9940b7060 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc9917fa640 1 -- 172.21.15.161:0/2979034340 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98c03e7e0 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.644+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9940b4240 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.645+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9940b7580 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.646+0000 7fc9917fa640 1 -- 172.21.15.161:0/2979034340 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc98c03e940 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.646+0000 7fc9917fa640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc97806b810 0x7fc97806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.646+0000 7fc9917fa640 1 -- 172.21.15.161:0/2979034340 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fc98c0acf40 con 0x7fc99415e400 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.646+0000 7fc998d8b640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc97806b810 0x7fc97806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.644 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.647+0000 7fc998d8b640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc97806b810 0x7fc97806dcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc98800aac0 tx=0x7fc988008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.645 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.650+0000 7fc9917fa640 1 -- 172.21.15.161:0/2979034340 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc98c077b80 con 0x7fc99415e400 2024-03-23T14:37:01.822 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.826+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc9940b7580 con 0x7fc99415e400 2024-03-23T14:37:01.823 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.827+0000 7fc9917fa640 1 -- 172.21.15.161:0/2979034340 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7fc9940b7580 con 0x7fc99415e400 2024-03-23T14:37:01.823 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:01.823 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:01.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc97806b810 msgr2=0x7fc97806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc97806b810 0x7fc97806dcd0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc98800aac0 tx=0x7fc988008040 comp rx=0 tx=0).stop 2024-03-23T14:37:01.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415e400 msgr2=0x7fc9940b8e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415e400 0x7fc9940b8e20 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fc98c036040 tx=0x7fc98c004a40 comp rx=0 tx=0).stop 2024-03-23T14:37:01.825 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 shutdown_connections 2024-03-23T14:37:01.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc97806b810 0x7fc97806dcd0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc99415e400 0x7fc9940b8e20 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc9940d1470 0x7fc99415d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 --2- 172.21.15.161:0/2979034340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc9940cf9d0 0x7fc9940b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 >> 172.21.15.161:0/2979034340 conn(0x7fc99409d190 msgr2=0x7fc9940b1680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:01.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 shutdown_connections 2024-03-23T14:37:01.826 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.829+0000 7fc99afef640 1 -- 172.21.15.161:0/2979034340 wait complete. 2024-03-23T14:37:01.837 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-23T14:37:01.837 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:37:01.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.941+0000 7fe459639640 1 Processor -- start 2024-03-23T14:37:01.937 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.941+0000 7fe459639640 1 -- start start 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe459639640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe4541690f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe459639640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4540701e0 con 0x7fe4540d0f80 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe459639640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4540d0af0 con 0x7fe4540cdf10 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe459639640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe4540cfee0 con 0x7fe4540cec90 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe4527fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe4540d0f80 0x7fe4540ce580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42874/0 (socket says 172.21.15.161:42874) 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe4527fc640 1 -- 172.21.15.161:0/3185599 learned_addr learned my addr 172.21.15.161:0/3185599 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe4537fe640 1 --2- 172.21.15.161:0/3185599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe4541690f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe451ffb640 1 -- 172.21.15.161:0/3185599 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2364353368 0 0) 0x7fe4540701e0 con 0x7fe4540d0f80 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe451ffb640 1 -- 172.21.15.161:0/3185599 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe43c003520 con 0x7fe4540d0f80 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe451ffb640 1 -- 172.21.15.161:0/3185599 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4274321245 0 0) 0x7fe4540d0af0 con 0x7fe4540cdf10 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.942+0000 7fe451ffb640 1 -- 172.21.15.161:0/3185599 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe4540701e0 con 0x7fe4540cdf10 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.943+0000 7fe4537fe640 1 -- 172.21.15.161:0/3185599 >> v1:172.21.15.183:6789/0 conn(0x7fe4540cdf10 legacy=0x7fe45406f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.939 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.943+0000 7fe4537fe640 1 -- 172.21.15.161:0/3185599 >> v1:172.21.15.161:6789/0 conn(0x7fe4540d0f80 legacy=0x7fe4540ce580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.943+0000 7fe4537fe640 1 -- 172.21.15.161:0/3185599 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4540d01c0 con 0x7fe4540cec90 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.943+0000 7fe4537fe640 1 --2- 172.21.15.161:0/3185599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe4541690f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe444009dc0 tx=0x7fe44402bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5d7addb2e52ffca server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.943+0000 7fe451ffb640 1 -- 172.21.15.161:0/3185599 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44402ccd0 con 0x7fe4540cec90 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe451ffb640 1 -- 172.21.15.161:0/3185599 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe44402ce30 con 0x7fe4540cec90 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe451ffb640 1 -- 172.21.15.161:0/3185599 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe444034660 con 0x7fe4540cec90 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe459639640 1 -- 172.21.15.161:0/3185599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 msgr2=0x7fe4541690f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe459639640 1 --2- 172.21.15.161:0/3185599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe4541690f0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe444009dc0 tx=0x7fe44402bf20 comp rx=0 tx=0).stop 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe459639640 1 -- 172.21.15.161:0/3185599 shutdown_connections 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe459639640 1 --2- 172.21.15.161:0/3185599 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe4541690f0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe459639640 1 -- 172.21.15.161:0/3185599 >> 172.21.15.161:0/3185599 conn(0x7fe4540d1d00 msgr2=0x7fe4540d2110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe459639640 1 -- 172.21.15.161:0/3185599 shutdown_connections 2024-03-23T14:37:01.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.944+0000 7fe459639640 1 -- 172.21.15.161:0/3185599 wait complete. 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.945+0000 7fe459639640 1 Processor -- start 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.945+0000 7fe459639640 1 -- start start 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe459639640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4540cdf10 0x7fe4541f75a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe459639640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe45415a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe459639640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4540d0f80 0x7fe454154600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe459639640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe454153e90 con 0x7fe4540d0f80 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe459639640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe45415cf50 con 0x7fe4540cdf10 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe459639640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe45415d230 con 0x7fe4540cec90 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4537fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4540d0f80 0x7fe454154600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4537fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4540d0f80 0x7fe454154600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45078/0 (socket says 172.21.15.161:45078) 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4537fe640 1 -- 172.21.15.161:0/3116128905 learned_addr learned my addr 172.21.15.161:0/3116128905 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4527fc640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe45415a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe452ffd640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4540cdf10 0x7fe4541f75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4537fe640 1 -- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 msgr2=0x7fe45415a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4537fe640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe45415a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4537fe640 1 -- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4540cdf10 msgr2=0x7fe4541f75a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4537fe640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4540cdf10 0x7fe4541f75a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.946+0000 7fe4537fe640 1 -- 172.21.15.161:0/3116128905 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe444009a70 con 0x7fe4540d0f80 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.947+0000 7fe452ffd640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4540cdf10 0x7fe4541f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.947+0000 7fe4527fc640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe45415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.947+0000 7fe4537fe640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4540d0f80 0x7fe454154600 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe44402dfd0 tx=0x7fe444004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.947+0000 7fe43b7fe640 1 -- 172.21.15.161:0/3116128905 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44403d600 con 0x7fe4540d0f80 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.947+0000 7fe43b7fe640 1 -- 172.21.15.161:0/3116128905 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe44403d760 con 0x7fe4540d0f80 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.947+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe45415d510 con 0x7fe4540d0f80 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.947+0000 7fe43b7fe640 1 -- 172.21.15.161:0/3116128905 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe44403db10 con 0x7fe4540d0f80 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.947+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4541f7ae0 con 0x7fe4540d0f80 2024-03-23T14:37:01.945 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.948+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4541f7c70 con 0x7fe4540d0f80 2024-03-23T14:37:01.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.948+0000 7fe43b7fe640 1 -- 172.21.15.161:0/3116128905 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe44403b040 con 0x7fe4540d0f80 2024-03-23T14:37:01.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.949+0000 7fe43b7fe640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe44006b840 0x7fe44006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:01.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.949+0000 7fe43b7fe640 1 -- 172.21.15.161:0/3116128905 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fe4440ac290 con 0x7fe4540d0f80 2024-03-23T14:37:01.946 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.949+0000 7fe452ffd640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe44006b840 0x7fe44006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:01.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.949+0000 7fe452ffd640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe44006b840 0x7fe44006dd00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe454149900 tx=0x7fe44c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:01.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:01.953+0000 7fe43b7fe640 1 -- 172.21.15.161:0/3116128905 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe444076d40 con 0x7fe4540d0f80 2024-03-23T14:37:02.236 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.240+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe4540d0780 con 0x7fe44006b840 2024-03-23T14:37:02.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.247+0000 7fe43b7fe640 1 -- 172.21.15.161:0/3116128905 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+237193 (secure 0 0 0) 0x7fe44c001ce0 con 0x7fe44006b840 2024-03-23T14:37:02.245 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:02.247 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe44006b840 msgr2=0x7fe44006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe44006b840 0x7fe44006dd00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe454149900 tx=0x7fe44c009040 comp rx=0 tx=0).stop 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4540d0f80 msgr2=0x7fe454154600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4540d0f80 0x7fe454154600 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fe44402dfd0 tx=0x7fe444004a40 comp rx=0 tx=0).stop 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 shutdown_connections 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe44006b840 0x7fe44006dd00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4540d0f80 0x7fe454154600 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4540cec90 0x7fe45415a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 --2- 172.21.15.161:0/3116128905 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4540cdf10 0x7fe4541f75a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.250+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 >> 172.21.15.161:0/3116128905 conn(0x7fe4540d1d00 msgr2=0x7fe4540c6e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.251+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 shutdown_connections 2024-03-23T14:37:02.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.251+0000 7fe459639640 1 -- 172.21.15.161:0/3116128905 wait complete. 2024-03-23T14:37:02.261 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":101,"stamp":"2024-03-23T14:37:00.494767+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":39,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":221216,"kb_used_data":5044,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754753504,"statfs":{"total":773094113280,"available":772867588096,"internally_reserved":0,"allocated":5165056,"data_stored":2796381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001017"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.15","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.14","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.13","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.10","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.7","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:58.145376+0000","last_change":"2024-03-23T14:36:58.145376+0000","last_active":"2024-03-23T14:36:58.145376+0000","last_peered":"2024-03-23T14:36:58.145376+0000","last_clean":"2024-03-23T14:36:58.145376+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:58.145376+0000","last_undegraded":"2024-03-23T14:36:58.145376+0000","last_fullsized":"2024-03-23T14:36:58.145376+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"7.6","version":"0'0","reported_seq":0,"reported_epoch":0,"state":"unknown","last_fresh":"2024-03-23T14:36:57.137879+0000","last_change":"2024-03-23T14:36:57.137879+0000","last_active":"2024-03-23T14:36:57.137879+0000","last_peered":"2024-03-23T14:36:57.137879+0000","last_clean":"2024-03-23T14:36:57.137879+0000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-03-23T14:36:57.137879+0000","last_undegraded":"2024-03-23T14:36:57.137879+0000","last_fullsized":"2024-03-23T14:36:57.137879+0000","mapping_epoch":0,"log_start":"0'0","ondisk_log_start":"0'0","created":0,"last_epoch_clean":0,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"--","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[],"acting":[],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":-1,"acting_primary":-1,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":207,"reported_epoch":41,"state":"active+clean","last_fresh":"2024-03-23T14:36:54.133038+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:36:54.133038+0000","last_peered":"2024-03-23T14:36:54.133038+0000","last_clean":"2024-03-23T14:36:54.133038+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:36:54.133038+0000","last_undegraded":"2024-03-23T14:36:54.133038+0000","last_fullsized":"2024-03-23T14:36:54.133038+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574875,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28076,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343764,"statfs":{"total":96636764160,"available":96608014336,"internally_reserved":0,"allocated":1077248,"data_stored":786576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27528,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344312,"statfs":{"total":96636764160,"available":96608575488,"internally_reserved":0,"allocated":520192,"data_stored":211535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344360,"statfs":{"total":96636764160,"available":96608624640,"internally_reserved":0,"allocated":471040,"data_stored":191065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574876,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27504,"kb_used_data":484,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344336,"statfs":{"total":96636764160,"available":96608600064,"internally_reserved":0,"allocated":495616,"data_stored":201351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27492,"kb_used_data":472,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344348,"statfs":{"total":96636764160,"available":96608612352,"internally_reserved":0,"allocated":483328,"data_stored":196208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27532,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":520192,"data_stored":211535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574876,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27532,"kb_used_data":508,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344308,"statfs":{"total":96636764160,"available":96608571392,"internally_reserved":0,"allocated":520192,"data_stored":211535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574875,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343768,"statfs":{"total":96636764160,"available":96608018432,"internally_reserved":0,"allocated":1077248,"data_stored":786576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:37:02.266 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:37:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.407+0000 7f5cbfa9d640 1 Processor -- start 2024-03-23T14:37:02.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.407+0000 7f5cbfa9d640 1 -- start start 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbfa9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5cb815e400 0x7f5cb80cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbfa9d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb80d0460 con 0x7f5cb80d2b40 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbfa9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cb80d06b0 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbfa9d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5cb80d1050 con 0x7f5cb815e400 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbd839640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5cb815e400 0x7f5cb80cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbd839640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5cb815e400 0x7f5cb80cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45082/0 (socket says 172.21.15.161:45082) 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbd839640 1 -- 172.21.15.161:0/1838890342 learned_addr learned my addr 172.21.15.161:0/1838890342 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbe03a640 1 --1- 172.21.15.161:0/1838890342 >> v1:172.21.15.161:6789/0 conn(0x7f5cb80d2b40 0x7f5cb80b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42876/0 (socket says 172.21.15.161:42876) 2024-03-23T14:37:02.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbd839640 1 -- 172.21.15.161:0/1838890342 >> v1:172.21.15.183:6789/0 conn(0x7f5cb80cf9d0 legacy=0x7f5cb80d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:02.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbc837640 1 -- 172.21.15.161:0/1838890342 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2505693810 0 0) 0x7f5cb80d0460 con 0x7f5cb80d2b40 2024-03-23T14:37:02.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbd839640 1 -- 172.21.15.161:0/1838890342 >> v1:172.21.15.161:6789/0 conn(0x7f5cb80d2b40 legacy=0x7f5cb80b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.408+0000 7f5cbd839640 1 -- 172.21.15.161:0/1838890342 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb80d0990 con 0x7f5cb815e400 2024-03-23T14:37:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.409+0000 7f5cbd839640 1 --2- 172.21.15.161:0/1838890342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb815e400 0x7f5cb80cf490 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009870 tx=0x7f5cb402ba20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e815b940b4174e57 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.409+0000 7f5cbc837640 1 -- 172.21.15.161:0/1838890342 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb402f840 con 0x7f5cb815e400 2024-03-23T14:37:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.409+0000 7f5cbc837640 1 -- 172.21.15.161:0/1838890342 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb402f9a0 con 0x7f5cb815e400 2024-03-23T14:37:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.409+0000 7f5cbc837640 1 -- 172.21.15.161:0/1838890342 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb402fd30 con 0x7f5cb815e400 2024-03-23T14:37:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.410+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1838890342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb815e400 msgr2=0x7f5cb80cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.410+0000 7f5cbfa9d640 1 --2- 172.21.15.161:0/1838890342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb815e400 0x7f5cb80cf490 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009870 tx=0x7f5cb402ba20 comp rx=0 tx=0).stop 2024-03-23T14:37:02.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.410+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1838890342 shutdown_connections 2024-03-23T14:37:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.410+0000 7f5cbfa9d640 1 --2- 172.21.15.161:0/1838890342 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb815e400 0x7f5cb80cf490 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.410+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1838890342 >> 172.21.15.161:0/1838890342 conn(0x7f5cb809d190 msgr2=0x7f5cb80d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.410+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1838890342 shutdown_connections 2024-03-23T14:37:02.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.410+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1838890342 wait complete. 2024-03-23T14:37:02.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.411+0000 7f5cbfa9d640 1 Processor -- start 2024-03-23T14:37:02.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.411+0000 7f5cbfa9d640 1 -- start start 2024-03-23T14:37:02.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbfa9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb80cf9d0 0x7f5cb80cc630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbfa9d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cb80d2b40 0x7f5cb8153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbfa9d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb815e400 0x7f5cb80b0e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbfa9d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb80cbea0 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbfa9d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5cb80ccb70 con 0x7f5cb815e400 2024-03-23T14:37:02.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbfa9d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5cb8153b90 con 0x7f5cb80d2b40 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb80cf9d0 0x7f5cb80cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb80cf9d0 0x7f5cb80cc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45098/0 (socket says 172.21.15.161:45098) 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 -- 172.21.15.161:0/1749367910 learned_addr learned my addr 172.21.15.161:0/1749367910 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbe03a640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb815e400 0x7f5cb80b0e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd038640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cb80d2b40 0x7f5cb8153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 -- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cb80d2b40 msgr2=0x7f5cb8153650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cb80d2b40 0x7f5cb8153650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 -- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb815e400 msgr2=0x7f5cb80b0e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb815e400 0x7f5cb80b0e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 -- 172.21.15.161:0/1749367910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cb40094d0 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbd839640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb80cf9d0 0x7f5cb80cc630 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40090b0 tx=0x7f5cb4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cae7fc640 1 -- 172.21.15.161:0/1749367910 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb403c040 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cae7fc640 1 -- 172.21.15.161:0/1749367910 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb4037d70 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cb80cb290 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cae7fc640 1 -- 172.21.15.161:0/1749367910 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4004270 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.412+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cb80cb850 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.414+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb80a1e00 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.414+0000 7f5cae7fc640 1 -- 172.21.15.161:0/1749367910 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5cb40043d0 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.414+0000 7f5cae7fc640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5c9c06b810 0x7f5c9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.414+0000 7f5cae7fc640 1 -- 172.21.15.161:0/1749367910 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5cb40ac700 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.415+0000 7f5cbd038640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5c9c06b810 0x7f5c9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.415+0000 7f5cbd038640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5c9c06b810 0x7f5c9c06dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80cdf20 tx=0x7f5ca8009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:02.414 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.418+0000 7f5cae7fc640 1 -- 172.21.15.161:0/1749367910 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5cb4077120 con 0x7f5cb80cf9d0 2024-03-23T14:37:02.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.593+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5cb8161680 con 0x7f5c9c06b810 2024-03-23T14:37:02.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.600+0000 7f5cae7fc640 1 -- 172.21.15.161:0/1749367910 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251957 (secure 0 0 0) 0x7f5cb8161680 con 0x7f5c9c06b810 2024-03-23T14:37:02.599 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:02.601 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:37:02.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5c9c06b810 msgr2=0x7f5c9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5c9c06b810 0x7f5c9c06dcd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f5cb80cdf20 tx=0x7f5ca8009040 comp rx=0 tx=0).stop 2024-03-23T14:37:02.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb80cf9d0 msgr2=0x7f5cb80cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb80cf9d0 0x7f5cb80cc630 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f5cb40090b0 tx=0x7f5cb4004730 comp rx=0 tx=0).stop 2024-03-23T14:37:02.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 shutdown_connections 2024-03-23T14:37:02.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5c9c06b810 0x7f5c9c06dcd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5cb815e400 0x7f5cb80b0e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5cb80d2b40 0x7f5cb8153650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 --2- 172.21.15.161:0/1749367910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5cb80cf9d0 0x7f5cb80cc630 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 >> 172.21.15.161:0/1749367910 conn(0x7f5cb809d190 msgr2=0x7f5cb8161a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:02.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 shutdown_connections 2024-03-23T14:37:02.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.604+0000 7f5cbfa9d640 1 -- 172.21.15.161:0/1749367910 wait complete. 2024-03-23T14:37:02.611 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":102,"stamp":"2024-03-23T14:37:02.495213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":24,"num_object_clones":0,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":204,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":156,"ondisk_log_size":156,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":180,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":221832,"kb_used_data":5624,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752888,"statfs":{"total":773094113280,"available":772866957312,"internally_reserved":0,"allocated":5758976,"data_stored":3053972,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":31,"apply_latency_ms":31,"commit_latency_ns":31000000,"apply_latency_ns":31000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.346397"},"pg_stats":[{"pgid":"8.3f","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175907+0000","last_change":"2024-03-23T14:36:59.178569+0000","last_active":"2024-03-23T14:37:00.175907+0000","last_peered":"2024-03-23T14:37:00.175907+0000","last_clean":"2024-03-23T14:37:00.175907+0000","last_became_active":"2024-03-23T14:36:59.178001+0000","last_became_peered":"2024-03-23T14:36:59.178001+0000","last_unstale":"2024-03-23T14:37:00.175907+0000","last_undegraded":"2024-03-23T14:37:00.175907+0000","last_fullsized":"2024-03-23T14:37:00.175907+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:52:17.139824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.3e","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305065+0000","last_change":"2024-03-23T14:36:59.178878+0000","last_active":"2024-03-23T14:37:00.305065+0000","last_peered":"2024-03-23T14:37:00.305065+0000","last_clean":"2024-03-23T14:37:00.305065+0000","last_became_active":"2024-03-23T14:36:59.178577+0000","last_became_peered":"2024-03-23T14:36:59.178577+0000","last_unstale":"2024-03-23T14:37:00.305065+0000","last_undegraded":"2024-03-23T14:37:00.305065+0000","last_fullsized":"2024-03-23T14:37:00.305065+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:45.916432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3d","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304677+0000","last_change":"2024-03-23T14:36:59.178721+0000","last_active":"2024-03-23T14:37:00.304677+0000","last_peered":"2024-03-23T14:37:00.304677+0000","last_clean":"2024-03-23T14:37:00.304677+0000","last_became_active":"2024-03-23T14:36:59.176880+0000","last_became_peered":"2024-03-23T14:36:59.176880+0000","last_unstale":"2024-03-23T14:37:00.304677+0000","last_undegraded":"2024-03-23T14:37:00.304677+0000","last_fullsized":"2024-03-23T14:37:00.304677+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:41:56.569747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.3c","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304735+0000","last_change":"2024-03-23T14:36:59.172182+0000","last_active":"2024-03-23T14:37:00.304735+0000","last_peered":"2024-03-23T14:37:00.304735+0000","last_clean":"2024-03-23T14:37:00.304735+0000","last_became_active":"2024-03-23T14:36:59.171942+0000","last_became_peered":"2024-03-23T14:36:59.171942+0000","last_unstale":"2024-03-23T14:37:00.304735+0000","last_undegraded":"2024-03-23T14:37:00.304735+0000","last_fullsized":"2024-03-23T14:37:00.304735+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:26.633373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.3b","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176015+0000","last_change":"2024-03-23T14:36:59.180526+0000","last_active":"2024-03-23T14:37:00.176015+0000","last_peered":"2024-03-23T14:37:00.176015+0000","last_clean":"2024-03-23T14:37:00.176015+0000","last_became_active":"2024-03-23T14:36:59.180216+0000","last_became_peered":"2024-03-23T14:36:59.180216+0000","last_unstale":"2024-03-23T14:37:00.176015+0000","last_undegraded":"2024-03-23T14:37:00.176015+0000","last_fullsized":"2024-03-23T14:37:00.176015+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:05:46.876289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.3a","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175776+0000","last_change":"2024-03-23T14:36:59.178846+0000","last_active":"2024-03-23T14:37:00.175776+0000","last_peered":"2024-03-23T14:37:00.175776+0000","last_clean":"2024-03-23T14:37:00.175776+0000","last_became_active":"2024-03-23T14:36:59.178573+0000","last_became_peered":"2024-03-23T14:36:59.178573+0000","last_unstale":"2024-03-23T14:37:00.175776+0000","last_undegraded":"2024-03-23T14:37:00.175776+0000","last_fullsized":"2024-03-23T14:37:00.175776+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:50.154093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.39","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304091+0000","last_change":"2024-03-23T14:36:59.178914+0000","last_active":"2024-03-23T14:37:00.304091+0000","last_peered":"2024-03-23T14:37:00.304091+0000","last_clean":"2024-03-23T14:37:00.304091+0000","last_became_active":"2024-03-23T14:36:59.176324+0000","last_became_peered":"2024-03-23T14:36:59.176324+0000","last_unstale":"2024-03-23T14:37:00.304091+0000","last_undegraded":"2024-03-23T14:37:00.304091+0000","last_fullsized":"2024-03-23T14:37:00.304091+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:13.267934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.38","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303326+0000","last_change":"2024-03-23T14:36:59.177909+0000","last_active":"2024-03-23T14:37:00.303326+0000","last_peered":"2024-03-23T14:37:00.303326+0000","last_clean":"2024-03-23T14:37:00.303326+0000","last_became_active":"2024-03-23T14:36:59.177684+0000","last_became_peered":"2024-03-23T14:36:59.177684+0000","last_unstale":"2024-03-23T14:37:00.303326+0000","last_undegraded":"2024-03-23T14:37:00.303326+0000","last_fullsized":"2024-03-23T14:37:00.303326+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:05:03.596800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.37","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303727+0000","last_change":"2024-03-23T14:36:59.181996+0000","last_active":"2024-03-23T14:37:00.303727+0000","last_peered":"2024-03-23T14:37:00.303727+0000","last_clean":"2024-03-23T14:37:00.303727+0000","last_became_active":"2024-03-23T14:36:59.181727+0000","last_became_peered":"2024-03-23T14:36:59.181727+0000","last_unstale":"2024-03-23T14:37:00.303727+0000","last_undegraded":"2024-03-23T14:37:00.303727+0000","last_fullsized":"2024-03-23T14:37:00.303727+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:13:57.961168+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.36","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175720+0000","last_change":"2024-03-23T14:36:59.173937+0000","last_active":"2024-03-23T14:37:00.175720+0000","last_peered":"2024-03-23T14:37:00.175720+0000","last_clean":"2024-03-23T14:37:00.175720+0000","last_became_active":"2024-03-23T14:36:59.173057+0000","last_became_peered":"2024-03-23T14:36:59.173057+0000","last_unstale":"2024-03-23T14:37:00.175720+0000","last_undegraded":"2024-03-23T14:37:00.175720+0000","last_fullsized":"2024-03-23T14:37:00.175720+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:31.687833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.35","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175824+0000","last_change":"2024-03-23T14:36:59.180333+0000","last_active":"2024-03-23T14:37:00.175824+0000","last_peered":"2024-03-23T14:37:00.175824+0000","last_clean":"2024-03-23T14:37:00.175824+0000","last_became_active":"2024-03-23T14:36:59.180020+0000","last_became_peered":"2024-03-23T14:36:59.180020+0000","last_unstale":"2024-03-23T14:37:00.175824+0000","last_undegraded":"2024-03-23T14:37:00.175824+0000","last_fullsized":"2024-03-23T14:37:00.175824+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:51:54.380014+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.34","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303729+0000","last_change":"2024-03-23T14:36:59.183588+0000","last_active":"2024-03-23T14:37:00.303729+0000","last_peered":"2024-03-23T14:37:00.303729+0000","last_clean":"2024-03-23T14:37:00.303729+0000","last_became_active":"2024-03-23T14:36:59.183342+0000","last_became_peered":"2024-03-23T14:36:59.183342+0000","last_unstale":"2024-03-23T14:37:00.303729+0000","last_undegraded":"2024-03-23T14:37:00.303729+0000","last_fullsized":"2024-03-23T14:37:00.303729+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:30:29.618046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.33","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175985+0000","last_change":"2024-03-23T14:36:59.174494+0000","last_active":"2024-03-23T14:37:00.175985+0000","last_peered":"2024-03-23T14:37:00.175985+0000","last_clean":"2024-03-23T14:37:00.175985+0000","last_became_active":"2024-03-23T14:36:59.174252+0000","last_became_peered":"2024-03-23T14:36:59.174252+0000","last_unstale":"2024-03-23T14:37:00.175985+0000","last_undegraded":"2024-03-23T14:37:00.175985+0000","last_fullsized":"2024-03-23T14:37:00.175985+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:36:35.025786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.32","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304618+0000","last_change":"2024-03-23T14:36:59.179453+0000","last_active":"2024-03-23T14:37:00.304618+0000","last_peered":"2024-03-23T14:37:00.304618+0000","last_clean":"2024-03-23T14:37:00.304618+0000","last_became_active":"2024-03-23T14:36:59.179194+0000","last_became_peered":"2024-03-23T14:36:59.179194+0000","last_unstale":"2024-03-23T14:37:00.304618+0000","last_undegraded":"2024-03-23T14:37:00.304618+0000","last_fullsized":"2024-03-23T14:37:00.304618+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:33.588969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.31","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304796+0000","last_change":"2024-03-23T14:36:59.185450+0000","last_active":"2024-03-23T14:37:00.304796+0000","last_peered":"2024-03-23T14:37:00.304796+0000","last_clean":"2024-03-23T14:37:00.304796+0000","last_became_active":"2024-03-23T14:36:59.185091+0000","last_became_peered":"2024-03-23T14:36:59.185091+0000","last_unstale":"2024-03-23T14:37:00.304796+0000","last_undegraded":"2024-03-23T14:37:00.304796+0000","last_fullsized":"2024-03-23T14:37:00.304796+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:46:14.769647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.30","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175443+0000","last_change":"2024-03-23T14:36:59.173708+0000","last_active":"2024-03-23T14:37:00.175443+0000","last_peered":"2024-03-23T14:37:00.175443+0000","last_clean":"2024-03-23T14:37:00.175443+0000","last_became_active":"2024-03-23T14:36:59.173453+0000","last_became_peered":"2024-03-23T14:36:59.173453+0000","last_unstale":"2024-03-23T14:37:00.175443+0000","last_undegraded":"2024-03-23T14:37:00.175443+0000","last_fullsized":"2024-03-23T14:37:00.175443+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:28.982567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.2f","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305090+0000","last_change":"2024-03-23T14:36:59.179460+0000","last_active":"2024-03-23T14:37:00.305090+0000","last_peered":"2024-03-23T14:37:00.305090+0000","last_clean":"2024-03-23T14:37:00.305090+0000","last_became_active":"2024-03-23T14:36:59.179165+0000","last_became_peered":"2024-03-23T14:36:59.179165+0000","last_unstale":"2024-03-23T14:37:00.305090+0000","last_undegraded":"2024-03-23T14:37:00.305090+0000","last_fullsized":"2024-03-23T14:37:00.305090+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:57:32.873857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2e","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305141+0000","last_change":"2024-03-23T14:36:59.178159+0000","last_active":"2024-03-23T14:37:00.305141+0000","last_peered":"2024-03-23T14:37:00.305141+0000","last_clean":"2024-03-23T14:37:00.305141+0000","last_became_active":"2024-03-23T14:36:59.177879+0000","last_became_peered":"2024-03-23T14:36:59.177879+0000","last_unstale":"2024-03-23T14:37:00.305141+0000","last_undegraded":"2024-03-23T14:37:00.305141+0000","last_fullsized":"2024-03-23T14:37:00.305141+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:53.301364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.2d","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304806+0000","last_change":"2024-03-23T14:36:59.178586+0000","last_active":"2024-03-23T14:37:00.304806+0000","last_peered":"2024-03-23T14:37:00.304806+0000","last_clean":"2024-03-23T14:37:00.304806+0000","last_became_active":"2024-03-23T14:36:59.176609+0000","last_became_peered":"2024-03-23T14:36:59.176609+0000","last_unstale":"2024-03-23T14:37:00.304806+0000","last_undegraded":"2024-03-23T14:37:00.304806+0000","last_fullsized":"2024-03-23T14:37:00.304806+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:45.916432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.2c","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304365+0000","last_change":"2024-03-23T14:36:59.180016+0000","last_active":"2024-03-23T14:37:00.304365+0000","last_peered":"2024-03-23T14:37:00.304365+0000","last_clean":"2024-03-23T14:37:00.304365+0000","last_became_active":"2024-03-23T14:36:59.179702+0000","last_became_peered":"2024-03-23T14:36:59.179702+0000","last_unstale":"2024-03-23T14:37:00.304365+0000","last_undegraded":"2024-03-23T14:37:00.304365+0000","last_fullsized":"2024-03-23T14:37:00.304365+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:45.406357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2b","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303963+0000","last_change":"2024-03-23T14:36:59.182705+0000","last_active":"2024-03-23T14:37:00.303963+0000","last_peered":"2024-03-23T14:37:00.303963+0000","last_clean":"2024-03-23T14:37:00.303963+0000","last_became_active":"2024-03-23T14:36:59.182304+0000","last_became_peered":"2024-03-23T14:36:59.182304+0000","last_unstale":"2024-03-23T14:37:00.303963+0000","last_undegraded":"2024-03-23T14:37:00.303963+0000","last_fullsized":"2024-03-23T14:37:00.303963+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:15:29.292181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.2a","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303382+0000","last_change":"2024-03-23T14:36:59.178569+0000","last_active":"2024-03-23T14:37:00.303382+0000","last_peered":"2024-03-23T14:37:00.303382+0000","last_clean":"2024-03-23T14:37:00.303382+0000","last_became_active":"2024-03-23T14:36:59.178309+0000","last_became_peered":"2024-03-23T14:36:59.178309+0000","last_unstale":"2024-03-23T14:37:00.303382+0000","last_undegraded":"2024-03-23T14:37:00.303382+0000","last_fullsized":"2024-03-23T14:37:00.303382+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:56:33.177393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.29","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175554+0000","last_change":"2024-03-23T14:36:59.170634+0000","last_active":"2024-03-23T14:37:00.175554+0000","last_peered":"2024-03-23T14:37:00.175554+0000","last_clean":"2024-03-23T14:37:00.175554+0000","last_became_active":"2024-03-23T14:36:59.170379+0000","last_became_peered":"2024-03-23T14:36:59.170379+0000","last_unstale":"2024-03-23T14:37:00.175554+0000","last_undegraded":"2024-03-23T14:37:00.175554+0000","last_fullsized":"2024-03-23T14:37:00.175554+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:12:37.185460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.28","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303898+0000","last_change":"2024-03-23T14:36:59.176430+0000","last_active":"2024-03-23T14:37:00.303898+0000","last_peered":"2024-03-23T14:37:00.303898+0000","last_clean":"2024-03-23T14:37:00.303898+0000","last_became_active":"2024-03-23T14:36:59.176173+0000","last_became_peered":"2024-03-23T14:36:59.176173+0000","last_unstale":"2024-03-23T14:37:00.303898+0000","last_undegraded":"2024-03-23T14:37:00.303898+0000","last_fullsized":"2024-03-23T14:37:00.303898+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:16:11.209373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.27","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305148+0000","last_change":"2024-03-23T14:36:59.176141+0000","last_active":"2024-03-23T14:37:00.305148+0000","last_peered":"2024-03-23T14:37:00.305148+0000","last_clean":"2024-03-23T14:37:00.305148+0000","last_became_active":"2024-03-23T14:36:59.175842+0000","last_became_peered":"2024-03-23T14:36:59.175842+0000","last_unstale":"2024-03-23T14:37:00.305148+0000","last_undegraded":"2024-03-23T14:37:00.305148+0000","last_fullsized":"2024-03-23T14:37:00.305148+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:09:36.960788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.26","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303960+0000","last_change":"2024-03-23T14:36:59.179500+0000","last_active":"2024-03-23T14:37:00.303960+0000","last_peered":"2024-03-23T14:37:00.303960+0000","last_clean":"2024-03-23T14:37:00.303960+0000","last_became_active":"2024-03-23T14:36:59.179253+0000","last_became_peered":"2024-03-23T14:36:59.179253+0000","last_unstale":"2024-03-23T14:37:00.303960+0000","last_undegraded":"2024-03-23T14:37:00.303960+0000","last_fullsized":"2024-03-23T14:37:00.303960+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:43:00.455216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.25","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176434+0000","last_change":"2024-03-23T14:36:59.183903+0000","last_active":"2024-03-23T14:37:00.176434+0000","last_peered":"2024-03-23T14:37:00.176434+0000","last_clean":"2024-03-23T14:37:00.176434+0000","last_became_active":"2024-03-23T14:36:59.183558+0000","last_became_peered":"2024-03-23T14:36:59.183558+0000","last_unstale":"2024-03-23T14:37:00.176434+0000","last_undegraded":"2024-03-23T14:37:00.176434+0000","last_fullsized":"2024-03-23T14:37:00.176434+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:55:08.380354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.24","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304524+0000","last_change":"2024-03-23T14:36:59.184109+0000","last_active":"2024-03-23T14:37:00.304524+0000","last_peered":"2024-03-23T14:37:00.304524+0000","last_clean":"2024-03-23T14:37:00.304524+0000","last_became_active":"2024-03-23T14:36:59.183568+0000","last_became_peered":"2024-03-23T14:36:59.183568+0000","last_unstale":"2024-03-23T14:37:00.304524+0000","last_undegraded":"2024-03-23T14:37:00.304524+0000","last_fullsized":"2024-03-23T14:37:00.304524+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:23:47.182240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.23","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304069+0000","last_change":"2024-03-23T14:36:59.189784+0000","last_active":"2024-03-23T14:37:00.304069+0000","last_peered":"2024-03-23T14:37:00.304069+0000","last_clean":"2024-03-23T14:37:00.304069+0000","last_became_active":"2024-03-23T14:36:59.189519+0000","last_became_peered":"2024-03-23T14:36:59.189519+0000","last_unstale":"2024-03-23T14:37:00.304069+0000","last_undegraded":"2024-03-23T14:37:00.304069+0000","last_fullsized":"2024-03-23T14:37:00.304069+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:45:05.295269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.22","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304748+0000","last_change":"2024-03-23T14:36:59.171468+0000","last_active":"2024-03-23T14:37:00.304748+0000","last_peered":"2024-03-23T14:37:00.304748+0000","last_clean":"2024-03-23T14:37:00.304748+0000","last_became_active":"2024-03-23T14:36:59.171213+0000","last_became_peered":"2024-03-23T14:36:59.171213+0000","last_unstale":"2024-03-23T14:37:00.304748+0000","last_undegraded":"2024-03-23T14:37:00.304748+0000","last_fullsized":"2024-03-23T14:37:00.304748+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:33.588969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.21","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304223+0000","last_change":"2024-03-23T14:36:59.178797+0000","last_active":"2024-03-23T14:37:00.304223+0000","last_peered":"2024-03-23T14:37:00.304223+0000","last_clean":"2024-03-23T14:37:00.304223+0000","last_became_active":"2024-03-23T14:36:59.176568+0000","last_became_peered":"2024-03-23T14:36:59.176568+0000","last_unstale":"2024-03-23T14:37:00.304223+0000","last_undegraded":"2024-03-23T14:37:00.304223+0000","last_fullsized":"2024-03-23T14:37:00.304223+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:00:26.633373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.20","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304188+0000","last_change":"2024-03-23T14:36:59.179949+0000","last_active":"2024-03-23T14:37:00.304188+0000","last_peered":"2024-03-23T14:37:00.304188+0000","last_clean":"2024-03-23T14:37:00.304188+0000","last_became_active":"2024-03-23T14:36:59.179462+0000","last_became_peered":"2024-03-23T14:36:59.179462+0000","last_unstale":"2024-03-23T14:37:00.304188+0000","last_undegraded":"2024-03-23T14:37:00.304188+0000","last_fullsized":"2024-03-23T14:37:00.304188+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:37:20.905185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.10","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303883+0000","last_change":"2024-03-23T14:36:59.179345+0000","last_active":"2024-03-23T14:37:00.303883+0000","last_peered":"2024-03-23T14:37:00.303883+0000","last_clean":"2024-03-23T14:37:00.303883+0000","last_became_active":"2024-03-23T14:36:59.178786+0000","last_became_peered":"2024-03-23T14:36:59.178786+0000","last_unstale":"2024-03-23T14:37:00.303883+0000","last_undegraded":"2024-03-23T14:37:00.303883+0000","last_fullsized":"2024-03-23T14:37:00.303883+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:37:13.267934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1f","version":"47'4","reported_seq":18,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304202+0000","last_change":"2024-03-23T14:36:58.166409+0000","last_active":"2024-03-23T14:37:00.304202+0000","last_peered":"2024-03-23T14:37:00.304202+0000","last_clean":"2024-03-23T14:37:00.304202+0000","last_became_active":"2024-03-23T14:36:58.166019+0000","last_became_peered":"2024-03-23T14:36:58.166019+0000","last_unstale":"2024-03-23T14:37:00.304202+0000","last_undegraded":"2024-03-23T14:37:00.304202+0000","last_fullsized":"2024-03-23T14:37:00.304202+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:00:54.670172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.11","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305103+0000","last_change":"2024-03-23T14:36:59.181258+0000","last_active":"2024-03-23T14:37:00.305103+0000","last_peered":"2024-03-23T14:37:00.305103+0000","last_clean":"2024-03-23T14:37:00.305103+0000","last_became_active":"2024-03-23T14:36:59.180804+0000","last_became_peered":"2024-03-23T14:36:59.180804+0000","last_unstale":"2024-03-23T14:37:00.305103+0000","last_undegraded":"2024-03-23T14:37:00.305103+0000","last_fullsized":"2024-03-23T14:37:00.305103+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:20.635074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1e","version":"47'1","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175825+0000","last_change":"2024-03-23T14:36:58.170778+0000","last_active":"2024-03-23T14:37:00.175825+0000","last_peered":"2024-03-23T14:37:00.175825+0000","last_clean":"2024-03-23T14:37:00.175825+0000","last_became_active":"2024-03-23T14:36:58.170401+0000","last_became_peered":"2024-03-23T14:36:58.170401+0000","last_unstale":"2024-03-23T14:37:00.175825+0000","last_undegraded":"2024-03-23T14:37:00.175825+0000","last_fullsized":"2024-03-23T14:37:00.175825+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:10:11.979003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.12","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304002+0000","last_change":"2024-03-23T14:36:59.184476+0000","last_active":"2024-03-23T14:37:00.304002+0000","last_peered":"2024-03-23T14:37:00.304002+0000","last_clean":"2024-03-23T14:37:00.304002+0000","last_became_active":"2024-03-23T14:36:59.184184+0000","last_became_peered":"2024-03-23T14:36:59.184184+0000","last_unstale":"2024-03-23T14:37:00.304002+0000","last_undegraded":"2024-03-23T14:37:00.304002+0000","last_fullsized":"2024-03-23T14:37:00.304002+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:19:54.876994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1d","version":"47'2","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176444+0000","last_change":"2024-03-23T14:36:58.166897+0000","last_active":"2024-03-23T14:37:00.176444+0000","last_peered":"2024-03-23T14:37:00.176444+0000","last_clean":"2024-03-23T14:37:00.176444+0000","last_became_active":"2024-03-23T14:36:58.166655+0000","last_became_peered":"2024-03-23T14:36:58.166655+0000","last_unstale":"2024-03-23T14:37:00.176444+0000","last_undegraded":"2024-03-23T14:37:00.176444+0000","last_fullsized":"2024-03-23T14:37:00.176444+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:15:28.284684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.13","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303365+0000","last_change":"2024-03-23T14:36:59.173970+0000","last_active":"2024-03-23T14:37:00.303365+0000","last_peered":"2024-03-23T14:37:00.303365+0000","last_clean":"2024-03-23T14:37:00.303365+0000","last_became_active":"2024-03-23T14:36:59.173685+0000","last_became_peered":"2024-03-23T14:36:59.173685+0000","last_unstale":"2024-03-23T14:37:00.303365+0000","last_undegraded":"2024-03-23T14:37:00.303365+0000","last_fullsized":"2024-03-23T14:37:00.303365+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:42.338228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.1c","version":"47'2","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304963+0000","last_change":"2024-03-23T14:36:58.163705+0000","last_active":"2024-03-23T14:37:00.304963+0000","last_peered":"2024-03-23T14:37:00.304963+0000","last_clean":"2024-03-23T14:37:00.304963+0000","last_became_active":"2024-03-23T14:36:58.163352+0000","last_became_peered":"2024-03-23T14:36:58.163352+0000","last_unstale":"2024-03-23T14:37:00.304963+0000","last_undegraded":"2024-03-23T14:37:00.304963+0000","last_fullsized":"2024-03-23T14:37:00.304963+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:29:26.414957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.14","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176162+0000","last_change":"2024-03-23T14:36:59.180363+0000","last_active":"2024-03-23T14:37:00.176162+0000","last_peered":"2024-03-23T14:37:00.176162+0000","last_clean":"2024-03-23T14:37:00.176162+0000","last_became_active":"2024-03-23T14:36:59.179789+0000","last_became_peered":"2024-03-23T14:36:59.179789+0000","last_unstale":"2024-03-23T14:37:00.176162+0000","last_undegraded":"2024-03-23T14:37:00.176162+0000","last_fullsized":"2024-03-23T14:37:00.176162+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:28.982567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1b","version":"47'1","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304137+0000","last_change":"2024-03-23T14:36:58.167966+0000","last_active":"2024-03-23T14:37:00.304137+0000","last_peered":"2024-03-23T14:37:00.304137+0000","last_clean":"2024-03-23T14:37:00.304137+0000","last_became_active":"2024-03-23T14:36:58.167522+0000","last_became_peered":"2024-03-23T14:36:58.167522+0000","last_unstale":"2024-03-23T14:37:00.304137+0000","last_undegraded":"2024-03-23T14:37:00.304137+0000","last_fullsized":"2024-03-23T14:37:00.304137+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:23:46.174743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.15","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304888+0000","last_change":"2024-03-23T14:36:59.178154+0000","last_active":"2024-03-23T14:37:00.304888+0000","last_peered":"2024-03-23T14:37:00.304888+0000","last_clean":"2024-03-23T14:37:00.304888+0000","last_became_active":"2024-03-23T14:36:59.177921+0000","last_became_peered":"2024-03-23T14:36:59.177921+0000","last_unstale":"2024-03-23T14:37:00.304888+0000","last_undegraded":"2024-03-23T14:37:00.304888+0000","last_fullsized":"2024-03-23T14:37:00.304888+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:41:56.569747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1a","version":"47'1","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175862+0000","last_change":"2024-03-23T14:36:58.165512+0000","last_active":"2024-03-23T14:37:00.175862+0000","last_peered":"2024-03-23T14:37:00.175862+0000","last_clean":"2024-03-23T14:37:00.175862+0000","last_became_active":"2024-03-23T14:36:58.165234+0000","last_became_peered":"2024-03-23T14:36:58.165234+0000","last_unstale":"2024-03-23T14:37:00.175862+0000","last_undegraded":"2024-03-23T14:37:00.175862+0000","last_fullsized":"2024-03-23T14:37:00.175862+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:19:53.869497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.16","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303797+0000","last_change":"2024-03-23T14:36:59.179567+0000","last_active":"2024-03-23T14:37:00.303797+0000","last_peered":"2024-03-23T14:37:00.303797+0000","last_clean":"2024-03-23T14:37:00.303797+0000","last_became_active":"2024-03-23T14:36:59.179314+0000","last_became_peered":"2024-03-23T14:36:59.179314+0000","last_unstale":"2024-03-23T14:37:00.303797+0000","last_undegraded":"2024-03-23T14:37:00.303797+0000","last_fullsized":"2024-03-23T14:37:00.303797+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:06:04.412789+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.19","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304108+0000","last_change":"2024-03-23T14:36:58.167034+0000","last_active":"2024-03-23T14:37:00.304108+0000","last_peered":"2024-03-23T14:37:00.304108+0000","last_clean":"2024-03-23T14:37:00.304108+0000","last_became_active":"2024-03-23T14:36:58.166707+0000","last_became_peered":"2024-03-23T14:36:58.166707+0000","last_unstale":"2024-03-23T14:37:00.304108+0000","last_undegraded":"2024-03-23T14:37:00.304108+0000","last_fullsized":"2024-03-23T14:37:00.304108+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:42:47.712198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.17","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304937+0000","last_change":"2024-03-23T14:36:59.177448+0000","last_active":"2024-03-23T14:37:00.304937+0000","last_peered":"2024-03-23T14:37:00.304937+0000","last_clean":"2024-03-23T14:37:00.304937+0000","last_became_active":"2024-03-23T14:36:59.177114+0000","last_became_peered":"2024-03-23T14:36:59.177114+0000","last_unstale":"2024-03-23T14:37:00.304937+0000","last_undegraded":"2024-03-23T14:37:00.304937+0000","last_fullsized":"2024-03-23T14:37:00.304937+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:38:48.430093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.18","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304008+0000","last_change":"2024-03-23T14:36:58.166965+0000","last_active":"2024-03-23T14:37:00.304008+0000","last_peered":"2024-03-23T14:37:00.304008+0000","last_clean":"2024-03-23T14:37:00.304008+0000","last_became_active":"2024-03-23T14:36:58.166667+0000","last_became_peered":"2024-03-23T14:36:58.166667+0000","last_unstale":"2024-03-23T14:37:00.304008+0000","last_undegraded":"2024-03-23T14:37:00.304008+0000","last_fullsized":"2024-03-23T14:37:00.304008+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:19:53.869497+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.18","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175786+0000","last_change":"2024-03-23T14:36:59.178494+0000","last_active":"2024-03-23T14:37:00.175786+0000","last_peered":"2024-03-23T14:37:00.175786+0000","last_clean":"2024-03-23T14:37:00.175786+0000","last_became_active":"2024-03-23T14:36:59.178241+0000","last_became_peered":"2024-03-23T14:36:59.178241+0000","last_unstale":"2024-03-23T14:37:00.175786+0000","last_undegraded":"2024-03-23T14:37:00.175786+0000","last_fullsized":"2024-03-23T14:37:00.175786+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:57:50.802614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304886+0000","last_change":"2024-03-23T14:36:58.162433+0000","last_active":"2024-03-23T14:37:00.304886+0000","last_peered":"2024-03-23T14:37:00.304886+0000","last_clean":"2024-03-23T14:37:00.304886+0000","last_became_active":"2024-03-23T14:36:58.162174+0000","last_became_peered":"2024-03-23T14:36:58.162174+0000","last_unstale":"2024-03-23T14:37:00.304886+0000","last_undegraded":"2024-03-23T14:37:00.304886+0000","last_fullsized":"2024-03-23T14:37:00.304886+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:05:30.680336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.19","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304198+0000","last_change":"2024-03-23T14:36:59.177102+0000","last_active":"2024-03-23T14:37:00.304198+0000","last_peered":"2024-03-23T14:37:00.304198+0000","last_clean":"2024-03-23T14:37:00.304198+0000","last_became_active":"2024-03-23T14:36:59.176778+0000","last_became_peered":"2024-03-23T14:36:59.176778+0000","last_unstale":"2024-03-23T14:37:00.304198+0000","last_undegraded":"2024-03-23T14:37:00.304198+0000","last_fullsized":"2024-03-23T14:37:00.304198+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:10.727936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176193+0000","last_change":"2024-03-23T14:36:58.168031+0000","last_active":"2024-03-23T14:37:00.176193+0000","last_peered":"2024-03-23T14:37:00.176193+0000","last_clean":"2024-03-23T14:37:00.176193+0000","last_became_active":"2024-03-23T14:36:58.167649+0000","last_became_peered":"2024-03-23T14:36:58.167649+0000","last_unstale":"2024-03-23T14:37:00.176193+0000","last_undegraded":"2024-03-23T14:37:00.176193+0000","last_fullsized":"2024-03-23T14:37:00.176193+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:03:12.304586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.1a","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304661+0000","last_change":"2024-03-23T14:36:59.188854+0000","last_active":"2024-03-23T14:37:00.304661+0000","last_peered":"2024-03-23T14:37:00.304661+0000","last_clean":"2024-03-23T14:37:00.304661+0000","last_became_active":"2024-03-23T14:36:59.188619+0000","last_became_peered":"2024-03-23T14:36:59.188619+0000","last_unstale":"2024-03-23T14:37:00.304661+0000","last_undegraded":"2024-03-23T14:37:00.304661+0000","last_fullsized":"2024-03-23T14:37:00.304661+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:46:14.769647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.15","version":"47'3","reported_seq":17,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305068+0000","last_change":"2024-03-23T14:36:58.165665+0000","last_active":"2024-03-23T14:37:00.305068+0000","last_peered":"2024-03-23T14:37:00.305068+0000","last_clean":"2024-03-23T14:37:00.305068+0000","last_became_active":"2024-03-23T14:36:58.165370+0000","last_became_peered":"2024-03-23T14:36:58.165370+0000","last_unstale":"2024-03-23T14:37:00.305068+0000","last_undegraded":"2024-03-23T14:37:00.305068+0000","last_fullsized":"2024-03-23T14:37:00.305068+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:37:19.897688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1b","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176062+0000","last_change":"2024-03-23T14:36:59.177648+0000","last_active":"2024-03-23T14:37:00.176062+0000","last_peered":"2024-03-23T14:37:00.176062+0000","last_clean":"2024-03-23T14:37:00.176062+0000","last_became_active":"2024-03-23T14:36:59.177360+0000","last_became_peered":"2024-03-23T14:36:59.177360+0000","last_unstale":"2024-03-23T14:37:00.176062+0000","last_undegraded":"2024-03-23T14:37:00.176062+0000","last_fullsized":"2024-03-23T14:37:00.176062+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:11:30.056048+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.14","version":"47'6","reported_seq":20,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303874+0000","last_change":"2024-03-23T14:36:58.169207+0000","last_active":"2024-03-23T14:37:00.303874+0000","last_peered":"2024-03-23T14:37:00.303874+0000","last_clean":"2024-03-23T14:37:00.303874+0000","last_became_active":"2024-03-23T14:36:58.168942+0000","last_became_peered":"2024-03-23T14:36:58.168942+0000","last_unstale":"2024-03-23T14:37:00.303874+0000","last_undegraded":"2024-03-23T14:37:00.303874+0000","last_fullsized":"2024-03-23T14:37:00.303874+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:36:34.018289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.1c","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304603+0000","last_change":"2024-03-23T14:36:59.180897+0000","last_active":"2024-03-23T14:37:00.304603+0000","last_peered":"2024-03-23T14:37:00.304603+0000","last_clean":"2024-03-23T14:37:00.304603+0000","last_became_active":"2024-03-23T14:36:59.180588+0000","last_became_peered":"2024-03-23T14:36:59.180588+0000","last_unstale":"2024-03-23T14:37:00.304603+0000","last_undegraded":"2024-03-23T14:37:00.304603+0000","last_fullsized":"2024-03-23T14:37:00.304603+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:52:17.139824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"47'2","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304189+0000","last_change":"2024-03-23T14:36:58.164773+0000","last_active":"2024-03-23T14:37:00.304189+0000","last_peered":"2024-03-23T14:37:00.304189+0000","last_clean":"2024-03-23T14:37:00.304189+0000","last_became_active":"2024-03-23T14:36:58.164515+0000","last_became_peered":"2024-03-23T14:36:58.164515+0000","last_unstale":"2024-03-23T14:37:00.304189+0000","last_undegraded":"2024-03-23T14:37:00.304189+0000","last_fullsized":"2024-03-23T14:37:00.304189+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:29:26.414957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1d","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304902+0000","last_change":"2024-03-23T14:36:59.178374+0000","last_active":"2024-03-23T14:37:00.304902+0000","last_peered":"2024-03-23T14:37:00.304902+0000","last_clean":"2024-03-23T14:37:00.304902+0000","last_became_active":"2024-03-23T14:36:59.176333+0000","last_became_peered":"2024-03-23T14:36:59.176333+0000","last_unstale":"2024-03-23T14:37:00.304902+0000","last_undegraded":"2024-03-23T14:37:00.304902+0000","last_fullsized":"2024-03-23T14:37:00.304902+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:24:53.301364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304103+0000","last_change":"2024-03-23T14:36:58.168591+0000","last_active":"2024-03-23T14:37:00.304103+0000","last_peered":"2024-03-23T14:37:00.304103+0000","last_clean":"2024-03-23T14:37:00.304103+0000","last_became_active":"2024-03-23T14:36:58.168385+0000","last_became_peered":"2024-03-23T14:36:58.168385+0000","last_unstale":"2024-03-23T14:37:00.304103+0000","last_undegraded":"2024-03-23T14:37:00.304103+0000","last_fullsized":"2024-03-23T14:37:00.304103+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:33:22.372347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.1e","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303655+0000","last_change":"2024-03-23T14:36:59.175572+0000","last_active":"2024-03-23T14:37:00.303655+0000","last_peered":"2024-03-23T14:37:00.303655+0000","last_clean":"2024-03-23T14:37:00.303655+0000","last_became_active":"2024-03-23T14:36:59.175320+0000","last_became_peered":"2024-03-23T14:36:59.175320+0000","last_unstale":"2024-03-23T14:37:00.303655+0000","last_undegraded":"2024-03-23T14:37:00.303655+0000","last_fullsized":"2024-03-23T14:37:00.303655+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:37:15.681897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304914+0000","last_change":"2024-03-23T14:36:58.166688+0000","last_active":"2024-03-23T14:37:00.304914+0000","last_peered":"2024-03-23T14:37:00.304914+0000","last_clean":"2024-03-23T14:37:00.304914+0000","last_became_active":"2024-03-23T14:36:58.166247+0000","last_became_peered":"2024-03-23T14:36:58.166247+0000","last_unstale":"2024-03-23T14:37:00.304914+0000","last_undegraded":"2024-03-23T14:37:00.304914+0000","last_fullsized":"2024-03-23T14:37:00.304914+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:49:27.975070+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.1f","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304984+0000","last_change":"2024-03-23T14:36:59.181708+0000","last_active":"2024-03-23T14:37:00.304984+0000","last_peered":"2024-03-23T14:37:00.304984+0000","last_clean":"2024-03-23T14:37:00.304984+0000","last_became_active":"2024-03-23T14:36:59.181412+0000","last_became_peered":"2024-03-23T14:36:59.181412+0000","last_unstale":"2024-03-23T14:37:00.304984+0000","last_undegraded":"2024-03-23T14:37:00.304984+0000","last_fullsized":"2024-03-23T14:37:00.304984+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:11:42.338228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"47'8","reported_seq":22,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303428+0000","last_change":"2024-03-23T14:36:58.170557+0000","last_active":"2024-03-23T14:37:00.303428+0000","last_peered":"2024-03-23T14:37:00.303428+0000","last_clean":"2024-03-23T14:37:00.303428+0000","last_became_active":"2024-03-23T14:36:58.170237+0000","last_became_peered":"2024-03-23T14:36:58.170237+0000","last_unstale":"2024-03-23T14:37:00.303428+0000","last_undegraded":"2024-03-23T14:37:00.303428+0000","last_fullsized":"2024-03-23T14:37:00.303428+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:19.627577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.0","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303405+0000","last_change":"2024-03-23T14:36:59.180357+0000","last_active":"2024-03-23T14:37:00.303405+0000","last_peered":"2024-03-23T14:37:00.303405+0000","last_clean":"2024-03-23T14:37:00.303405+0000","last_became_active":"2024-03-23T14:36:59.180024+0000","last_became_peered":"2024-03-23T14:36:59.180024+0000","last_unstale":"2024-03-23T14:37:00.303405+0000","last_undegraded":"2024-03-23T14:37:00.303405+0000","last_fullsized":"2024-03-23T14:37:00.303405+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:57:32.873857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.f","version":"47'1","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305143+0000","last_change":"2024-03-23T14:36:58.171064+0000","last_active":"2024-03-23T14:37:00.305143+0000","last_peered":"2024-03-23T14:37:00.305143+0000","last_clean":"2024-03-23T14:37:00.305143+0000","last_became_active":"2024-03-23T14:36:58.170770+0000","last_became_peered":"2024-03-23T14:36:58.170770+0000","last_unstale":"2024-03-23T14:37:00.305143+0000","last_undegraded":"2024-03-23T14:37:00.305143+0000","last_fullsized":"2024-03-23T14:37:00.305143+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:08:44.398860+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.1","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175762+0000","last_change":"2024-03-23T14:36:59.177449+0000","last_active":"2024-03-23T14:37:00.175762+0000","last_peered":"2024-03-23T14:37:00.175762+0000","last_clean":"2024-03-23T14:37:00.175762+0000","last_became_active":"2024-03-23T14:36:59.177176+0000","last_became_peered":"2024-03-23T14:36:59.177176+0000","last_unstale":"2024-03-23T14:37:00.175762+0000","last_undegraded":"2024-03-23T14:37:00.175762+0000","last_fullsized":"2024-03-23T14:37:00.175762+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:37:20.905185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.e","version":"47'2","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175955+0000","last_change":"2024-03-23T14:36:58.155773+0000","last_active":"2024-03-23T14:37:00.175955+0000","last_peered":"2024-03-23T14:37:00.175955+0000","last_clean":"2024-03-23T14:37:00.175955+0000","last_became_active":"2024-03-23T14:36:58.155514+0000","last_became_peered":"2024-03-23T14:36:58.155514+0000","last_unstale":"2024-03-23T14:37:00.175955+0000","last_undegraded":"2024-03-23T14:37:00.175955+0000","last_fullsized":"2024-03-23T14:37:00.175955+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:28:35.317812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.2","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175780+0000","last_change":"2024-03-23T14:36:59.181671+0000","last_active":"2024-03-23T14:37:00.175780+0000","last_peered":"2024-03-23T14:37:00.175780+0000","last_clean":"2024-03-23T14:37:00.175780+0000","last_became_active":"2024-03-23T14:36:59.181401+0000","last_became_peered":"2024-03-23T14:36:59.181401+0000","last_unstale":"2024-03-23T14:37:00.175780+0000","last_undegraded":"2024-03-23T14:37:00.175780+0000","last_fullsized":"2024-03-23T14:37:00.175780+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:53:23.232688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.d","version":"47'1","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304118+0000","last_change":"2024-03-23T14:36:58.170593+0000","last_active":"2024-03-23T14:37:00.304118+0000","last_peered":"2024-03-23T14:37:00.304118+0000","last_clean":"2024-03-23T14:37:00.304118+0000","last_became_active":"2024-03-23T14:36:58.170324+0000","last_became_peered":"2024-03-23T14:36:58.170324+0000","last_unstale":"2024-03-23T14:37:00.304118+0000","last_undegraded":"2024-03-23T14:37:00.304118+0000","last_fullsized":"2024-03-23T14:37:00.304118+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:55:30.492467+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1],"acting":[6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.3","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176124+0000","last_change":"2024-03-23T14:36:59.183259+0000","last_active":"2024-03-23T14:37:00.176124+0000","last_peered":"2024-03-23T14:37:00.176124+0000","last_clean":"2024-03-23T14:37:00.176124+0000","last_became_active":"2024-03-23T14:36:59.182929+0000","last_became_peered":"2024-03-23T14:36:59.182929+0000","last_unstale":"2024-03-23T14:37:00.176124+0000","last_undegraded":"2024-03-23T14:37:00.176124+0000","last_fullsized":"2024-03-23T14:37:00.176124+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:08.260209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.c","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304621+0000","last_change":"2024-03-23T14:36:58.170928+0000","last_active":"2024-03-23T14:37:00.304621+0000","last_peered":"2024-03-23T14:37:00.304621+0000","last_clean":"2024-03-23T14:37:00.304621+0000","last_became_active":"2024-03-23T14:36:58.170646+0000","last_became_peered":"2024-03-23T14:36:58.170646+0000","last_unstale":"2024-03-23T14:37:00.304621+0000","last_undegraded":"2024-03-23T14:37:00.304621+0000","last_fullsized":"2024-03-23T14:37:00.304621+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:59:42.009572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.4","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304951+0000","last_change":"2024-03-23T14:36:59.177452+0000","last_active":"2024-03-23T14:37:00.304951+0000","last_peered":"2024-03-23T14:37:00.304951+0000","last_clean":"2024-03-23T14:37:00.304951+0000","last_became_active":"2024-03-23T14:36:59.177140+0000","last_became_peered":"2024-03-23T14:36:59.177140+0000","last_unstale":"2024-03-23T14:37:00.304951+0000","last_undegraded":"2024-03-23T14:37:00.304951+0000","last_fullsized":"2024-03-23T14:37:00.304951+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:08.260209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.b","version":"47'2","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176115+0000","last_change":"2024-03-23T14:36:58.160359+0000","last_active":"2024-03-23T14:37:00.176115+0000","last_peered":"2024-03-23T14:37:00.176115+0000","last_clean":"2024-03-23T14:37:00.176115+0000","last_became_active":"2024-03-23T14:36:58.160094+0000","last_became_peered":"2024-03-23T14:36:58.160094+0000","last_unstale":"2024-03-23T14:37:00.176115+0000","last_undegraded":"2024-03-23T14:37:00.176115+0000","last_fullsized":"2024-03-23T14:37:00.176115+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T16:18:27.074232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"8.5","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303928+0000","last_change":"2024-03-23T14:36:59.182726+0000","last_active":"2024-03-23T14:37:00.303928+0000","last_peered":"2024-03-23T14:37:00.303928+0000","last_clean":"2024-03-23T14:37:00.303928+0000","last_became_active":"2024-03-23T14:36:59.182345+0000","last_became_peered":"2024-03-23T14:36:59.182345+0000","last_unstale":"2024-03-23T14:37:00.303928+0000","last_undegraded":"2024-03-23T14:37:00.303928+0000","last_fullsized":"2024-03-23T14:37:00.303928+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T20:10:12.986500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.a","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303271+0000","last_change":"2024-03-23T14:36:58.168309+0000","last_active":"2024-03-23T14:37:00.303271+0000","last_peered":"2024-03-23T14:37:00.303271+0000","last_clean":"2024-03-23T14:37:00.303271+0000","last_became_active":"2024-03-23T14:36:58.168021+0000","last_became_peered":"2024-03-23T14:36:58.168021+0000","last_unstale":"2024-03-23T14:37:00.303271+0000","last_undegraded":"2024-03-23T14:37:00.303271+0000","last_fullsized":"2024-03-23T14:37:00.303271+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:09.720439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.6","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304951+0000","last_change":"2024-03-23T14:36:59.188017+0000","last_active":"2024-03-23T14:37:00.304951+0000","last_peered":"2024-03-23T14:37:00.304951+0000","last_clean":"2024-03-23T14:37:00.304951+0000","last_became_active":"2024-03-23T14:36:59.187737+0000","last_became_peered":"2024-03-23T14:36:59.187737+0000","last_unstale":"2024-03-23T14:37:00.304951+0000","last_undegraded":"2024-03-23T14:37:00.304951+0000","last_fullsized":"2024-03-23T14:37:00.304951+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T22:56:33.177393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304304+0000","last_change":"2024-03-23T14:36:58.165669+0000","last_active":"2024-03-23T14:37:00.304304+0000","last_peered":"2024-03-23T14:37:00.304304+0000","last_clean":"2024-03-23T14:37:00.304304+0000","last_became_active":"2024-03-23T14:36:58.165434+0000","last_became_peered":"2024-03-23T14:36:58.165434+0000","last_unstale":"2024-03-23T14:37:00.304304+0000","last_undegraded":"2024-03-23T14:37:00.304304+0000","last_fullsized":"2024-03-23T14:37:00.304304+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:15:28.284684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.7","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.176272+0000","last_change":"2024-03-23T14:36:59.178787+0000","last_active":"2024-03-23T14:37:00.176272+0000","last_peered":"2024-03-23T14:37:00.176272+0000","last_clean":"2024-03-23T14:37:00.176272+0000","last_became_active":"2024-03-23T14:36:59.178318+0000","last_became_peered":"2024-03-23T14:36:59.178318+0000","last_unstale":"2024-03-23T14:37:00.176272+0000","last_undegraded":"2024-03-23T14:37:00.176272+0000","last_fullsized":"2024-03-23T14:37:00.176272+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:10:20.635074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175345+0000","last_change":"2024-03-23T14:36:58.160532+0000","last_active":"2024-03-23T14:37:00.175345+0000","last_peered":"2024-03-23T14:37:00.175345+0000","last_clean":"2024-03-23T14:37:00.175345+0000","last_became_active":"2024-03-23T14:36:58.160258+0000","last_became_peered":"2024-03-23T14:36:58.160258+0000","last_unstale":"2024-03-23T14:37:00.175345+0000","last_undegraded":"2024-03-23T14:37:00.175345+0000","last_fullsized":"2024-03-23T14:37:00.175345+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:55:03.961239+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"8.8","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304194+0000","last_change":"2024-03-23T14:36:59.179718+0000","last_active":"2024-03-23T14:37:00.304194+0000","last_peered":"2024-03-23T14:37:00.304194+0000","last_clean":"2024-03-23T14:37:00.304194+0000","last_became_active":"2024-03-23T14:36:59.179115+0000","last_became_peered":"2024-03-23T14:36:59.179115+0000","last_unstale":"2024-03-23T14:37:00.304194+0000","last_undegraded":"2024-03-23T14:37:00.304194+0000","last_fullsized":"2024-03-23T14:37:00.304194+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:50.154093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"47'2","reported_seq":16,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304316+0000","last_change":"2024-03-23T14:36:58.166869+0000","last_active":"2024-03-23T14:37:00.304316+0000","last_peered":"2024-03-23T14:37:00.304316+0000","last_clean":"2024-03-23T14:37:00.304316+0000","last_became_active":"2024-03-23T14:36:58.166510+0000","last_became_peered":"2024-03-23T14:36:58.166510+0000","last_unstale":"2024-03-23T14:37:00.304316+0000","last_undegraded":"2024-03-23T14:37:00.304316+0000","last_fullsized":"2024-03-23T14:37:00.304316+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:55:00.633109+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"8.a","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175906+0000","last_change":"2024-03-23T14:36:59.173425+0000","last_active":"2024-03-23T14:37:00.175906+0000","last_peered":"2024-03-23T14:37:00.175906+0000","last_clean":"2024-03-23T14:37:00.175906+0000","last_became_active":"2024-03-23T14:36:59.172792+0000","last_became_peered":"2024-03-23T14:36:59.172792+0000","last_unstale":"2024-03-23T14:37:00.175906+0000","last_undegraded":"2024-03-23T14:37:00.175906+0000","last_fullsized":"2024-03-23T14:37:00.175906+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:41:10.727936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305174+0000","last_change":"2024-03-23T14:36:58.163195+0000","last_active":"2024-03-23T14:37:00.305174+0000","last_peered":"2024-03-23T14:37:00.305174+0000","last_clean":"2024-03-23T14:37:00.305174+0000","last_became_active":"2024-03-23T14:36:58.162934+0000","last_became_peered":"2024-03-23T14:36:58.162934+0000","last_unstale":"2024-03-23T14:37:00.305174+0000","last_undegraded":"2024-03-23T14:37:00.305174+0000","last_fullsized":"2024-03-23T14:37:00.305174+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:13:49.146596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"8.b","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303514+0000","last_change":"2024-03-23T14:36:59.173623+0000","last_active":"2024-03-23T14:37:00.303514+0000","last_peered":"2024-03-23T14:37:00.303514+0000","last_clean":"2024-03-23T14:37:00.303514+0000","last_became_active":"2024-03-23T14:36:59.173346+0000","last_became_peered":"2024-03-23T14:36:59.173346+0000","last_unstale":"2024-03-23T14:37:00.303514+0000","last_undegraded":"2024-03-23T14:37:00.303514+0000","last_fullsized":"2024-03-23T14:37:00.303514+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T18:09:36.960788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304754+0000","last_change":"2024-03-23T14:36:58.167804+0000","last_active":"2024-03-23T14:37:00.304754+0000","last_peered":"2024-03-23T14:37:00.304754+0000","last_clean":"2024-03-23T14:37:00.304754+0000","last_became_active":"2024-03-23T14:36:58.165275+0000","last_became_peered":"2024-03-23T14:36:58.165275+0000","last_unstale":"2024-03-23T14:37:00.304754+0000","last_undegraded":"2024-03-23T14:37:00.304754+0000","last_fullsized":"2024-03-23T14:37:00.304754+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:03:12.304586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"8.c","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304776+0000","last_change":"2024-03-23T14:36:59.179008+0000","last_active":"2024-03-23T14:37:00.304776+0000","last_peered":"2024-03-23T14:37:00.304776+0000","last_clean":"2024-03-23T14:37:00.304776+0000","last_became_active":"2024-03-23T14:36:59.178415+0000","last_became_peered":"2024-03-23T14:36:59.178415+0000","last_unstale":"2024-03-23T14:37:00.304776+0000","last_undegraded":"2024-03-23T14:37:00.304776+0000","last_fullsized":"2024-03-23T14:37:00.304776+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:38:48.430093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.3","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303495+0000","last_change":"2024-03-23T14:36:58.163424+0000","last_active":"2024-03-23T14:37:00.303495+0000","last_peered":"2024-03-23T14:37:00.303495+0000","last_clean":"2024-03-23T14:37:00.303495+0000","last_became_active":"2024-03-23T14:36:58.163173+0000","last_became_peered":"2024-03-23T14:36:58.163173+0000","last_unstale":"2024-03-23T14:37:00.303495+0000","last_undegraded":"2024-03-23T14:37:00.303495+0000","last_fullsized":"2024-03-23T14:37:00.303495+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T15:05:07.252712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.d","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304841+0000","last_change":"2024-03-23T14:36:59.181038+0000","last_active":"2024-03-23T14:37:00.304841+0000","last_peered":"2024-03-23T14:37:00.304841+0000","last_clean":"2024-03-23T14:37:00.304841+0000","last_became_active":"2024-03-23T14:36:59.180701+0000","last_became_peered":"2024-03-23T14:36:59.180701+0000","last_unstale":"2024-03-23T14:37:00.304841+0000","last_undegraded":"2024-03-23T14:37:00.304841+0000","last_fullsized":"2024-03-23T14:37:00.304841+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:36:35.025786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303378+0000","last_change":"2024-03-23T14:36:58.165514+0000","last_active":"2024-03-23T14:37:00.303378+0000","last_peered":"2024-03-23T14:37:00.303378+0000","last_clean":"2024-03-23T14:37:00.303378+0000","last_became_active":"2024-03-23T14:36:58.165260+0000","last_became_peered":"2024-03-23T14:36:58.165260+0000","last_unstale":"2024-03-23T14:37:00.303378+0000","last_undegraded":"2024-03-23T14:37:00.303378+0000","last_fullsized":"2024-03-23T14:37:00.303378+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T23:52:16.132327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"8.e","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304908+0000","last_change":"2024-03-23T14:36:59.183374+0000","last_active":"2024-03-23T14:37:00.304908+0000","last_peered":"2024-03-23T14:37:00.304908+0000","last_clean":"2024-03-23T14:37:00.304908+0000","last_became_active":"2024-03-23T14:36:59.183128+0000","last_became_peered":"2024-03-23T14:36:59.183128+0000","last_unstale":"2024-03-23T14:37:00.304908+0000","last_undegraded":"2024-03-23T14:37:00.304908+0000","last_fullsized":"2024-03-23T14:37:00.304908+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:05:03.596800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.175763+0000","last_change":"2024-03-23T14:36:58.170777+0000","last_active":"2024-03-23T14:37:00.175763+0000","last_peered":"2024-03-23T14:37:00.175763+0000","last_clean":"2024-03-23T14:37:00.175763+0000","last_became_active":"2024-03-23T14:36:58.170440+0000","last_became_peered":"2024-03-23T14:36:58.170440+0000","last_unstale":"2024-03-23T14:37:00.175763+0000","last_undegraded":"2024-03-23T14:37:00.175763+0000","last_fullsized":"2024-03-23T14:37:00.175763+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:15:28.284684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"8.f","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.303858+0000","last_change":"2024-03-23T14:36:59.184352+0000","last_active":"2024-03-23T14:37:00.303858+0000","last_peered":"2024-03-23T14:37:00.303858+0000","last_clean":"2024-03-23T14:37:00.303858+0000","last_became_active":"2024-03-23T14:36:59.184110+0000","last_became_peered":"2024-03-23T14:36:59.184110+0000","last_unstale":"2024-03-23T14:37:00.303858+0000","last_undegraded":"2024-03-23T14:37:00.303858+0000","last_fullsized":"2024-03-23T14:37:00.303858+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:48:29.871840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.0","version":"0'0","reported_seq":14,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304310+0000","last_change":"2024-03-23T14:36:58.165037+0000","last_active":"2024-03-23T14:37:00.304310+0000","last_peered":"2024-03-23T14:37:00.304310+0000","last_clean":"2024-03-23T14:37:00.304310+0000","last_became_active":"2024-03-23T14:36:58.164756+0000","last_became_peered":"2024-03-23T14:36:58.164756+0000","last_unstale":"2024-03-23T14:37:00.304310+0000","last_undegraded":"2024-03-23T14:37:00.304310+0000","last_fullsized":"2024-03-23T14:37:00.304310+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T17:23:46.174743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"8.9","version":"0'0","reported_seq":12,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304316+0000","last_change":"2024-03-23T14:36:59.182857+0000","last_active":"2024-03-23T14:37:00.304316+0000","last_peered":"2024-03-23T14:37:00.304316+0000","last_clean":"2024-03-23T14:37:00.304316+0000","last_became_active":"2024-03-23T14:36:59.182562+0000","last_became_peered":"2024-03-23T14:36:59.182562+0000","last_unstale":"2024-03-23T14:37:00.304316+0000","last_undegraded":"2024-03-23T14:37:00.304316+0000","last_fullsized":"2024-03-23T14:37:00.304316+0000","mapping_epoch":46,"log_start":"0'0","ondisk_log_start":"0'0","created":46,"last_epoch_clean":47,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:58.145376+0000","last_clean_scrub_stamp":"2024-03-23T14:36:58.145376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T19:03:13.312083+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.6","version":"47'1","reported_seq":15,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.305013+0000","last_change":"2024-03-23T14:36:58.164851+0000","last_active":"2024-03-23T14:37:00.305013+0000","last_peered":"2024-03-23T14:37:00.305013+0000","last_clean":"2024-03-23T14:37:00.305013+0000","last_became_active":"2024-03-23T14:36:58.164530+0000","last_became_peered":"2024-03-23T14:36:58.164530+0000","last_unstale":"2024-03-23T14:37:00.305013+0000","last_undegraded":"2024-03-23T14:37:00.305013+0000","last_fullsized":"2024-03-23T14:37:00.305013+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":45,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:36:57.137879+0000","last_clean_scrub_stamp":"2024-03-23T14:36:57.137879+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T21:50:00.921295+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":219,"reported_epoch":48,"state":"active+clean","last_fresh":"2024-03-23T14:37:00.304415+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:37:00.304415+0000","last_peered":"2024-03-23T14:37:00.304415+0000","last_clean":"2024-03-23T14:37:00.304415+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:37:00.304415+0000","last_undegraded":"2024-03-23T14:37:00.304415+0000","last_fullsized":"2024-03-23T14:37:00.304415+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":8,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":7,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574876,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28156,"kb_used_data":1132,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343684,"statfs":{"total":96636764160,"available":96607932416,"internally_reserved":0,"allocated":1159168,"data_stored":820894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27576,"kb_used_data":552,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344264,"statfs":{"total":96636764160,"available":96608526336,"internally_reserved":0,"allocated":565248,"data_stored":231054,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27568,"kb_used_data":544,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344272,"statfs":{"total":96636764160,"available":96608534528,"internally_reserved":0,"allocated":557056,"data_stored":230402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574877,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27588,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344252,"statfs":{"total":96636764160,"available":96608514048,"internally_reserved":0,"allocated":573440,"data_stored":237284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27584,"kb_used_data":560,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344256,"statfs":{"total":96636764160,"available":96608518144,"internally_reserved":0,"allocated":573440,"data_stored":232036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574876,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27596,"kb_used_data":568,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344244,"statfs":{"total":96636764160,"available":96608505856,"internally_reserved":0,"allocated":581632,"data_stored":238314,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":564,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":577536,"data_stored":236862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574876,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28172,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343668,"statfs":{"total":96636764160,"available":96607916032,"internally_reserved":0,"allocated":1171456,"data_stored":827126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":56,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":652,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":8,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:37:02.615 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-23T14:37:02.615 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mon 'allow r' osd 'allow rw tag cephfs data=cephfs' mds allow 2024-03-23T14:37:02.762 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.765+0000 7f5b92d5e640 1 Processor -- start 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.765+0000 7f5b92d5e640 1 -- start start 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b92d5e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b8c0cfb90 0x7f5b8c0b8b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b92d5e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5b8c0d2a80 0x7f5b8c0b90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b92d5e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b8c0b86e0 con 0x7f5b8c15e230 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b92d5e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5b8c0b7ad0 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b92d5e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f5b8c0b7db0 con 0x7f5b8c0d2a80 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b8bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b8c0cfb90 0x7f5b8c0b8b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b8bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b8c0cfb90 0x7f5b8c0b8b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45108/0 (socket says 172.21.15.161:45108) 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b90afa640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5b8c15e230 0x7f5b8c0d2370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42892/0 (socket says 172.21.15.161:42892) 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b8bfff640 1 -- 172.21.15.161:0/284932453 learned_addr learned my addr 172.21.15.161:0/284932453 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b912fb640 1 --2- 172.21.15.161:0/284932453 >> v2:172.21.15.183:3300/0 conn(0x7f5b8c0d2a80 0x7f5b8c0b90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.763 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b8bfff640 1 -- 172.21.15.161:0/284932453 >> v2:172.21.15.183:3300/0 conn(0x7f5b8c0d2a80 msgr2=0x7f5b8c0b90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b8bfff640 1 --2- 172.21.15.161:0/284932453 >> v2:172.21.15.183:3300/0 conn(0x7f5b8c0d2a80 0x7f5b8c0b90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b8bfff640 1 -- 172.21.15.161:0/284932453 >> v1:172.21.15.161:6789/0 conn(0x7f5b8c15e230 legacy=0x7f5b8c0d2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b8b7fe640 1 -- 172.21.15.161:0/284932453 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1187171327 0 0) 0x7f5b8c0b86e0 con 0x7f5b8c15e230 2024-03-23T14:37:02.764 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.766+0000 7f5b8bfff640 1 -- 172.21.15.161:0/284932453 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b8c0b8090 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.767+0000 7f5b8bfff640 1 --2- 172.21.15.161:0/284932453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c0b8b70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c08e5c0 tx=0x7f5b8002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c64da09b0bb3834 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.767+0000 7f5b8b7fe640 1 -- 172.21.15.161:0/284932453 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80030860 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.767+0000 7f5b8b7fe640 1 -- 172.21.15.161:0/284932453 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b800309c0 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.767+0000 7f5b8b7fe640 1 -- 172.21.15.161:0/284932453 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b80030d50 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.768+0000 7f5b92d5e640 1 -- 172.21.15.161:0/284932453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 msgr2=0x7f5b8c0b8b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.768+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/284932453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c0b8b70 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f5b8c08e5c0 tx=0x7f5b8002b9a0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.768+0000 7f5b92d5e640 1 -- 172.21.15.161:0/284932453 shutdown_connections 2024-03-23T14:37:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.768+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/284932453 >> v2:172.21.15.183:3300/0 conn(0x7f5b8c0d2a80 0x7f5b8c0b90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.768+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/284932453 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c0b8b70 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.768+0000 7f5b92d5e640 1 -- 172.21.15.161:0/284932453 >> 172.21.15.161:0/284932453 conn(0x7f5b8c0d3e20 msgr2=0x7f5b8c0d4230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.768+0000 7f5b92d5e640 1 -- 172.21.15.161:0/284932453 shutdown_connections 2024-03-23T14:37:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.768+0000 7f5b92d5e640 1 -- 172.21.15.161:0/284932453 wait complete. 2024-03-23T14:37:02.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.769+0000 7f5b92d5e640 1 Processor -- start 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.769+0000 7f5b92d5e640 1 -- start start 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b92d5e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c1523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b92d5e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b8c0d2a80 0x7f5b8c1f4bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b92d5e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b8c15e230 0x7f5b8c09ff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b92d5e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c09e760 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b92d5e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5b8c1f5110 con 0x7f5b8c0d2a80 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b92d5e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5b8c09db20 con 0x7f5b8c15e230 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b90afa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c1523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b90afa640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c1523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45116/0 (socket says 172.21.15.161:45116) 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b90afa640 1 -- 172.21.15.161:0/1424241280 learned_addr learned my addr 172.21.15.161:0/1424241280 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b8bfff640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b8c0d2a80 0x7f5b8c1f4bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b90afa640 1 -- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b8c15e230 msgr2=0x7f5b8c09ff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b90afa640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b8c15e230 0x7f5b8c09ff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b90afa640 1 -- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b8c0d2a80 msgr2=0x7f5b8c1f4bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b90afa640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b8c0d2a80 0x7f5b8c1f4bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.770+0000 7f5b90afa640 1 -- 172.21.15.161:0/1424241280 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b800094d0 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.771+0000 7f5b90afa640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c1523c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400c7c0 tx=0x7f5b8400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.771+0000 7f5b897fa640 1 -- 172.21.15.161:0/1424241280 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400a640 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.771+0000 7f5b897fa640 1 -- 172.21.15.161:0/1424241280 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b8400a7c0 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.771+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b8c09ddc0 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.771+0000 7f5b897fa640 1 -- 172.21.15.161:0/1424241280 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b8400ab70 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.771+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b8c09e380 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.772+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b8c04f4e0 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.772+0000 7f5b897fa640 1 -- 172.21.15.161:0/1424241280 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5b8400acd0 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.773+0000 7f5b897fa640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b7806b810 0x7f5b7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.773+0000 7f5b897fa640 1 -- 172.21.15.161:0/1424241280 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5b84089a50 con 0x7f5b8c0cfb90 2024-03-23T14:37:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.773+0000 7f5b8bfff640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b7806b810 0x7f5b7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.774+0000 7f5b8bfff640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b7806b810 0x7f5b7806dcd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5b80036000 tx=0x7f5b8003b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:02.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:02.777+0000 7f5b897fa640 1 -- 172.21.15.161:0/1424241280 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5b84054470 con 0x7f5b8c0cfb90 2024-03-23T14:37:03.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.020+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]} v 0) -- 0x7f5b8c0b6470 con 0x7f5b8c0cfb90 2024-03-23T14:37:03.018 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.023+0000 7f5b897fa640 1 -- 172.21.15.161:0/1424241280 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mon", "allow r", "osd", "allow rw tag cephfs data=cephfs", "mds", "allow"]}]=0 updated caps for client.0 v15) ==== 190+0+147 (secure 0 0 0) 0x7f5b84018020 con 0x7f5b8c0cfb90 2024-03-23T14:37:03.019 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:37:03.019 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:37:03.019 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-23T14:37:03.019 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-23T14:37:03.019 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw tag cephfs data=cephfs" 2024-03-23T14:37:03.019 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b7806b810 msgr2=0x7f5b7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b7806b810 0x7f5b7806dcd0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f5b80036000 tx=0x7f5b8003b040 comp rx=0 tx=0).stop 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 msgr2=0x7f5b8c1523c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c1523c0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400c7c0 tx=0x7f5b8400cc80 comp rx=0 tx=0).stop 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 shutdown_connections 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b7806b810 0x7f5b7806dcd0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b8c15e230 0x7f5b8c09ff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b8c0d2a80 0x7f5b8c1f4bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.021 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 --2- 172.21.15.161:0/1424241280 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b8c0cfb90 0x7f5b8c1523c0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 >> 172.21.15.161:0/1424241280 conn(0x7f5b8c0d3e20 msgr2=0x7f5b8c06c430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 shutdown_connections 2024-03-23T14:37:03.022 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.025+0000 7f5b92d5e640 1 -- 172.21.15.161:0/1424241280 wait complete. 2024-03-23T14:37:03.032 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:37:03.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.137+0000 7f2ebad6c640 1 Processor -- start 2024-03-23T14:37:03.134 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.138+0000 7f2ebad6c640 1 -- start start 2024-03-23T14:37:03.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2ebad6c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2eb415e400 0x7f2eb40cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.135 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2ebad6c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2eb40cf9d0 0x7f2eb40d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2ebad6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40d1470 0x7f2eb40d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2ebad6c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb40d1050 con 0x7f2eb40cf9d0 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2ebad6c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f2eb40d0460 con 0x7f2eb415e400 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2ebad6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb40d06b0 con 0x7f2eb40d1470 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2eb40cf9d0 0x7f2eb40d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2eb40cf9d0 0x7f2eb40d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45130/0 (socket says 172.21.15.161:45130) 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb3fff640 1 -- 172.21.15.161:0/1643285790 learned_addr learned my addr 172.21.15.161:0/1643285790 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb9309640 1 --2- 172.21.15.161:0/1643285790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40d1470 0x7f2eb40d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb8b08640 1 --2- 172.21.15.161:0/1643285790 >> v2:172.21.15.183:3300/0 conn(0x7f2eb415e400 0x7f2eb40cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb3fff640 1 -- 172.21.15.161:0/1643285790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40d1470 msgr2=0x7f2eb40d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb3fff640 1 --2- 172.21.15.161:0/1643285790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40d1470 0x7f2eb40d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb3fff640 1 -- 172.21.15.161:0/1643285790 >> v2:172.21.15.183:3300/0 conn(0x7f2eb415e400 msgr2=0x7f2eb40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb3fff640 1 --2- 172.21.15.161:0/1643285790 >> v2:172.21.15.183:3300/0 conn(0x7f2eb415e400 0x7f2eb40cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb3fff640 1 -- 172.21.15.161:0/1643285790 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb40d0990 con 0x7f2eb40cf9d0 2024-03-23T14:37:03.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb9309640 1 --2- 172.21.15.161:0/1643285790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40d1470 0x7f2eb40d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:03.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.139+0000 7f2eb8b08640 1 --2- 172.21.15.161:0/1643285790 >> v2:172.21.15.183:3300/0 conn(0x7f2eb415e400 0x7f2eb40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:03.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.140+0000 7f2eb3fff640 1 --2- 172.21.15.161:0/1643285790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb40cf9d0 0x7f2eb40d2430 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2eb408e230 tx=0x7f2ea002b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8dbdebf328f9c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2eb37fe640 1 -- 172.21.15.161:0/1643285790 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0031870 con 0x7f2eb40cf9d0 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2eb37fe640 1 -- 172.21.15.161:0/1643285790 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ea00319d0 con 0x7f2eb40cf9d0 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2eb37fe640 1 -- 172.21.15.161:0/1643285790 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0031d60 con 0x7f2eb40cf9d0 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 -- 172.21.15.161:0/1643285790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb40cf9d0 msgr2=0x7f2eb40d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/1643285790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb40cf9d0 0x7f2eb40d2430 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f2eb408e230 tx=0x7f2ea002b910 comp rx=0 tx=0).stop 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 -- 172.21.15.161:0/1643285790 shutdown_connections 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/1643285790 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40d1470 0x7f2eb40d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/1643285790 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb40cf9d0 0x7f2eb40d2430 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/1643285790 >> v2:172.21.15.183:3300/0 conn(0x7f2eb415e400 0x7f2eb40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 -- 172.21.15.161:0/1643285790 >> 172.21.15.161:0/1643285790 conn(0x7f2eb409d190 msgr2=0x7f2eb40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 -- 172.21.15.161:0/1643285790 shutdown_connections 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.141+0000 7f2ebad6c640 1 -- 172.21.15.161:0/1643285790 wait complete. 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.142+0000 7f2ebad6c640 1 Processor -- start 2024-03-23T14:37:03.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.142+0000 7f2ebad6c640 1 -- start start 2024-03-23T14:37:03.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2ebad6c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40cf9d0 0x7f2eb415ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2ebad6c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb40d1470 0x7f2eb40c9510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2ebad6c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb415e400 0x7f2eb4161d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2ebad6c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb415fea0 con 0x7f2eb415e400 2024-03-23T14:37:03.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2ebad6c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb415f600 con 0x7f2eb40d1470 2024-03-23T14:37:03.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2ebad6c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2eb415f8e0 con 0x7f2eb40cf9d0 2024-03-23T14:37:03.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2eb9309640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb415e400 0x7f2eb4161d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.142 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2eb9309640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb415e400 0x7f2eb4161d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45134/0 (socket says 172.21.15.161:45134) 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.143+0000 7f2eb9309640 1 -- 172.21.15.161:0/2768754274 learned_addr learned my addr 172.21.15.161:0/2768754274 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb3fff640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb40d1470 0x7f2eb40c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb8b08640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40cf9d0 0x7f2eb415ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb9309640 1 -- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40cf9d0 msgr2=0x7f2eb415ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb9309640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40cf9d0 0x7f2eb415ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb9309640 1 -- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb40d1470 msgr2=0x7f2eb40c9510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb9309640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb40d1470 0x7f2eb40c9510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb9309640 1 -- 172.21.15.161:0/2768754274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ea00094d0 con 0x7f2eb415e400 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb3fff640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb40d1470 0x7f2eb40c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb9309640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb415e400 0x7f2eb4161d30 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00cef0 tx=0x7f2eac00a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb17fa640 1 -- 172.21.15.161:0/2768754274 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00d840 con 0x7f2eb415e400 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb17fa640 1 -- 172.21.15.161:0/2768754274 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2eac00d9a0 con 0x7f2eb415e400 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2eb17fa640 1 -- 172.21.15.161:0/2768754274 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00dd50 con 0x7f2eb415e400 2024-03-23T14:37:03.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb415fbc0 con 0x7f2eb415e400 2024-03-23T14:37:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.144+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb40b9510 con 0x7f2eb415e400 2024-03-23T14:37:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.145+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb40c7cf0 con 0x7f2eb415e400 2024-03-23T14:37:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.145+0000 7f2eb17fa640 1 -- 172.21.15.161:0/2768754274 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2eac01e3b0 con 0x7f2eb415e400 2024-03-23T14:37:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.146+0000 7f2eb17fa640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e9806b810 0x7f2e9806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.146+0000 7f2eb17fa640 1 -- 172.21.15.161:0/2768754274 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f2eac08abb0 con 0x7f2eb415e400 2024-03-23T14:37:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.146+0000 7f2eb8b08640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e9806b810 0x7f2e9806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.146+0000 7f2eb8b08640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e9806b810 0x7f2e9806dcd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2eb406c0e0 tx=0x7f2ea8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.146 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.150+0000 7f2eb17fa640 1 -- 172.21.15.161:0/2768754274 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2eb40c7cf0 con 0x7f2eb415e400 2024-03-23T14:37:03.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.369+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2eb40c7cf0 con 0x7f2eb415e400 2024-03-23T14:37:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.369+0000 7f2eb17fa640 1 -- 172.21.15.161:0/2768754274 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5193 (secure 0 0 0) 0x7f2eb40c7cf0 con 0x7f2eb415e400 2024-03-23T14:37:03.365 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:03.365 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2496234294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":2496234294},{"type":"v1","addr":"172.21.15.183:6836","nonce":2496234294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5396,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2645821131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2645821131},{"type":"v1","addr":"172.21.15.183:6837","nonce":2645821131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/2183619206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2183619206},{"type":"v1","addr":"172.21.15.161:6834","nonce":2183619206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:58.503230+0000","modified":"2024-03-23T14:37:00.601296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/1671955402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":1671955402},{"type":"v1","addr":"172.21.15.161:6835","nonce":1671955402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5925,"qdb_cluster":[5925]},"id":3}]} 2024-03-23T14:37:03.365 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-23T14:37:03.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e9806b810 msgr2=0x7f2e9806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e9806b810 0x7f2e9806dcd0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f2eb406c0e0 tx=0x7f2ea8008040 comp rx=0 tx=0).stop 2024-03-23T14:37:03.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb415e400 msgr2=0x7f2eb4161d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb415e400 0x7f2eb4161d30 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00cef0 tx=0x7f2eac00a710 comp rx=0 tx=0).stop 2024-03-23T14:37:03.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 shutdown_connections 2024-03-23T14:37:03.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f2e9806b810 0x7f2e9806dcd0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2eb415e400 0x7f2eb4161d30 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2eb40d1470 0x7f2eb40c9510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 --2- 172.21.15.161:0/2768754274 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2eb40cf9d0 0x7f2eb415ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 >> 172.21.15.161:0/2768754274 conn(0x7f2eb409d190 msgr2=0x7f2eb415d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 shutdown_connections 2024-03-23T14:37:03.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.371+0000 7f2ebad6c640 1 -- 172.21.15.161:0/2768754274 wait complete. 2024-03-23T14:37:03.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-23T14:36:58.503230+0000', 'modified': '2024-03-23T14:37:00.601296+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5925}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5925': {'gid': 5925, 'name': 'a', 'rank': 0, 'incarnation': 21, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.161:6835/1671955402', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.161:6833', 'nonce': 1671955402}, {'type': 'v1', 'addr': '172.21.15.161:6835', 'nonce': 1671955402}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 5925, 'qdb_cluster': [5925]} 2024-03-23T14:37:03.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-23T14:37:03.781 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a status 2024-03-23T14:37:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.889+0000 7f47e89af640 1 Processor -- start 2024-03-23T14:37:03.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.889+0000 7f47e89af640 1 -- start start 2024-03-23T14:37:03.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e89af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 0x7f47e40cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e89af640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47e40d2b40 0x7f47e40b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e89af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f47e40d0460 con 0x7f47e40cf9d0 2024-03-23T14:37:03.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e89af640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f47e40d1050 con 0x7f47e40d2b40 2024-03-23T14:37:03.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e89af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f47e40d06b0 con 0x7f47e415e400 2024-03-23T14:37:03.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47e40d2b40 0x7f47e40b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e2d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f47e40d2b40 0x7f47e40b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45138/0 (socket says 172.21.15.161:45138) 2024-03-23T14:37:03.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e2d9d640 1 -- 172.21.15.161:0/3531519605 learned_addr learned my addr 172.21.15.161:0/3531519605 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e259c640 1 --2- 172.21.15.161:0/3531519605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 0x7f47e40cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e2d9d640 1 -- 172.21.15.161:0/3531519605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 msgr2=0x7f47e40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/3531519605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 0x7f47e40cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e2d9d640 1 -- 172.21.15.161:0/3531519605 >> v1:172.21.15.183:6789/0 conn(0x7f47e40cf9d0 legacy=0x7f47e40d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.890+0000 7f47e2d9d640 1 -- 172.21.15.161:0/3531519605 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47e40d0990 con 0x7f47e40d2b40 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.891+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/3531519605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40d2b40 0x7f47e40b0a80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009870 tx=0x7f47dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f3787865094322 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.891+0000 7f47e1d9b640 1 -- 172.21.15.161:0/3531519605 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc030840 con 0x7f47e40d2b40 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.891+0000 7f47e1d9b640 1 -- 172.21.15.161:0/3531519605 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc0309a0 con 0x7f47e40d2b40 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.891+0000 7f47e1d9b640 1 -- 172.21.15.161:0/3531519605 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc030d30 con 0x7f47e40d2b40 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.892+0000 7f47e89af640 1 -- 172.21.15.161:0/3531519605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40d2b40 msgr2=0x7f47e40b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.892+0000 7f47e89af640 1 --2- 172.21.15.161:0/3531519605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40d2b40 0x7f47e40b0a80 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009870 tx=0x7f47dc02b980 comp rx=0 tx=0).stop 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.892+0000 7f47e89af640 1 -- 172.21.15.161:0/3531519605 shutdown_connections 2024-03-23T14:37:03.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.892+0000 7f47e89af640 1 --2- 172.21.15.161:0/3531519605 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40d2b40 0x7f47e40b0a80 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.892+0000 7f47e89af640 1 --2- 172.21.15.161:0/3531519605 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 0x7f47e40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.892+0000 7f47e89af640 1 -- 172.21.15.161:0/3531519605 >> 172.21.15.161:0/3531519605 conn(0x7f47e409d190 msgr2=0x7f47e40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.892+0000 7f47e89af640 1 -- 172.21.15.161:0/3531519605 shutdown_connections 2024-03-23T14:37:03.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.892+0000 7f47e89af640 1 -- 172.21.15.161:0/3531519605 wait complete. 2024-03-23T14:37:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.893+0000 7f47e89af640 1 Processor -- start 2024-03-23T14:37:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.893+0000 7f47e89af640 1 -- start start 2024-03-23T14:37:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e89af640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40cf9d0 0x7f47e415d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e89af640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f47e40d2b40 0x7f47e40b5670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e89af640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 0x7f47e40b5bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e89af640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f47e40b8300 con 0x7f47e40cf9d0 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e89af640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f47e40b8e40 con 0x7f47e40d2b40 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e89af640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f47e40b92d0 con 0x7f47e415e400 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40cf9d0 0x7f47e415d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40cf9d0 0x7f47e415d4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45154/0 (socket says 172.21.15.161:45154) 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e259c640 1 -- 172.21.15.161:0/3093384801 learned_addr learned my addr 172.21.15.161:0/3093384801 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f47e40d2b40 0x7f47e40b5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e259c640 1 -- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 msgr2=0x7f47e40b5bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e259c640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 0x7f47e40b5bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e259c640 1 -- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f47e40d2b40 msgr2=0x7f47e40b5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e259c640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f47e40d2b40 0x7f47e40b5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.894+0000 7f47e259c640 1 -- 172.21.15.161:0/3093384801 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47dc0094d0 con 0x7f47e40cf9d0 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.895+0000 7f47e259c640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40cf9d0 0x7f47e415d4a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f47d4007e60 tx=0x7f47d400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.895+0000 7f47db7fe640 1 -- 172.21.15.161:0/3093384801 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4014620 con 0x7f47e40cf9d0 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.895+0000 7f47db7fe640 1 -- 172.21.15.161:0/3093384801 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47d400ee90 con 0x7f47e40cf9d0 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.895+0000 7f47db7fe640 1 -- 172.21.15.161:0/3093384801 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d40149e0 con 0x7f47e40cf9d0 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.895+0000 7f47e89af640 1 -- 172.21.15.161:0/3093384801 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47e40b8510 con 0x7f47e40cf9d0 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.895+0000 7f47e89af640 1 -- 172.21.15.161:0/3093384801 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e40b8ad0 con 0x7f47e40cf9d0 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.896+0000 7f47db7fe640 1 -- 172.21.15.161:0/3093384801 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f47d4014b40 con 0x7f47e40cf9d0 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.897+0000 7f47db7fe640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f47c406b810 0x7f47c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.897+0000 7f47db7fe640 1 -- 172.21.15.161:0/3093384801 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f47d408ab60 con 0x7f47e40cf9d0 2024-03-23T14:37:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.897+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f47c406b810 0x7f47c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.898+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f47c406b810 0x7f47c406dcd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f47e41607d0 tx=0x7f47cc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.898+0000 7f47eb58c740 1 Processor -- start 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.898+0000 7f47eb58c740 1 -- start start 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 0x563baa1395b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa1693a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa139af0 0x563baa1698e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47eb58c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563baa0ad180 con 0x563baa139af0 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47eb58c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563baa0aa6b0 con 0x563baa166610 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47eb58c740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x563baa0a5e90 con 0x563baa1669f0 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa139af0 0x563baa1698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa139af0 0x563baa1698e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45162/0 (socket says 172.21.15.161:45162) 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 -- 172.21.15.161:0/1658318982 learned_addr learned my addr 172.21.15.161:0/1658318982 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e259c640 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 0x563baa1395b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa1693a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 -- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 msgr2=0x563baa1693a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa1693a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 -- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 msgr2=0x563baa1395b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 0x563baa1395b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 -- 172.21.15.161:0/1658318982 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563baa062d80 con 0x563baa139af0 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa1693a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa139af0 0x563baa1698e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009990 tx=0x7f47dc030000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb5a194be300950c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47c3fff640 1 -- 172.21.15.161:0/1658318982 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc030b10 con 0x563baa139af0 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47c3fff640 1 -- 172.21.15.161:0/1658318982 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc030c70 con 0x563baa139af0 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.899+0000 7f47c3fff640 1 -- 172.21.15.161:0/1658318982 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47dc044480 con 0x563baa139af0 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 -- 172.21.15.161:0/1658318982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa139af0 msgr2=0x563baa1698e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa139af0 0x563baa1698e0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f47dc009990 tx=0x7f47dc030000 comp rx=0 tx=0).stop 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 -- 172.21.15.161:0/1658318982 shutdown_connections 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa139af0 0x563baa1698e0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa1693a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 --2- 172.21.15.161:0/1658318982 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 0x563baa1395b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 -- 172.21.15.161:0/1658318982 >> 172.21.15.161:0/1658318982 conn(0x563baa13b820 msgr2=0x563baa1843a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 -- 172.21.15.161:0/1658318982 shutdown_connections 2024-03-23T14:37:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.900+0000 7f47eb58c740 1 -- 172.21.15.161:0/1658318982 wait complete. 2024-03-23T14:37:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.901+0000 7f47eb58c740 1 -- start start 2024-03-23T14:37:03.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.901+0000 7f47eb58c740 1 Processor -- start 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa166610 0x563baa164880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa164dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa170b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47eb58c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563baa04ad50 con 0x563baa166610 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47eb58c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563ba9d79310 con 0x563baa16e0c0 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47eb58c740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x563baa02b8f0 con 0x563baa1669f0 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47eb58c740 10 client.0 _ll_register_callbacks cb 0x563ba9db6ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa166610 0x563baa164880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa166610 0x563baa164880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45172/0 (socket says 172.21.15.161:45172) 2024-03-23T14:37:03.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 -- 172.21.15.161:0/2544851589 learned_addr learned my addr 172.21.15.161:0/2544851589 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa170b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa164dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 -- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 msgr2=0x563baa164dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa164dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 -- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 msgr2=0x563baa170b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa170b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 -- 172.21.15.161:0/2544851589 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e40d0990 con 0x563baa166610 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa170b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47e259c640 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa166610 0x563baa164880 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f47d400e9b0 tx=0x7f47d405d4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d26a45943456411a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47c17fa640 1 -- 172.21.15.161:0/2544851589 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47d4055950 con 0x563baa166610 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47c17fa640 1 -- 172.21.15.161:0/2544851589 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f47d4055ab0 con 0x563baa166610 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.903+0000 7f47eb58c740 1 -- 172.21.15.161:0/2544851589 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x563ba9da56c0 con 0x563baa166610 2024-03-23T14:37:03.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47c17fa640 1 -- 172.21.15.161:0/2544851589 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f47d4061560 con 0x563baa166610 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47eb58c740 10 client.6066 fetch_fsmap learned FSMap version 23 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47eb58c740 1 -- 172.21.15.161:0/2544851589 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x563baa0a4510 con 0x563baa166610 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47c17fa640 1 -- 172.21.15.161:0/2544851589 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x563ba9da56c0 con 0x563baa166610 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47c17fa640 1 -- 172.21.15.161:0/2544851589 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f47d4058730 con 0x563baa166610 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47eb58c740 10 client.6066 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47eb58c740 10 client.6066 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47eb58c740 20 client.6066 populate_metadata read hostname 'smithi161' 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47eb58c740 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] conn(0x563baa194640 0x563baa16b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47eb58c740 4 client.6066 mds_command: new command op to 5925 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47eb58c740 1 -- 172.21.15.161:0/2544851589 --> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ba9df6270 con 0x563baa194640 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] conn(0x563baa194640 0x563baa16b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] conn(0x563baa194640 0x563baa16b590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=456362ccea4a8262 server_cookie=719484b254d76fd9 in_seq=0 out_seq=0 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.904+0000 7f47c17fa640 10 client.6066 ms_handle_connect on v2:172.21.15.161:6833/1671955402 2024-03-23T14:37:03.906 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47c17fa640 1 -- 172.21.15.161:0/2544851589 <== mds.0 v2:172.21.15.161:6833/1671955402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+22966 (crc 0 0 0) 0x563ba9df6270 con 0x563baa194640 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47c17fa640 10 client.6066 handle_command_reply: tid=0 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 client.6066 shutdown 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 -- 172.21.15.161:0/2544851589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa166610 msgr2=0x563baa164880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa166610 0x563baa164880 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f47d400e9b0 tx=0x7f47d405d4f0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 -- 172.21.15.161:0/2544851589 shutdown_connections 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] conn(0x563baa194640 0x563baa16b590 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa170b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa1669f0 0x563baa164dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 --2- 172.21.15.161:0/2544851589 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa166610 0x563baa164880 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47c17fa640 0 client.6066 ms_handle_reset on v2:172.21.15.161:6833/1671955402 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 -- 172.21.15.161:0/2544851589 >> 172.21.15.161:0/2544851589 conn(0x563baa139af0 msgr2=0x563baa13d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 -- 172.21.15.161:0/2544851589 shutdown_connections 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 1 -- 172.21.15.161:0/2544851589 wait complete. 2024-03-23T14:37:03.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.905+0000 7f47eb58c740 20 client.6066 trim_cache size 0 max 16384 2024-03-23T14:37:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.923+0000 7f47eb58c740 1 Processor -- start 2024-03-23T14:37:03.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.923+0000 7f47eb58c740 1 -- start start 2024-03-23T14:37:03.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 0x563baa134ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa172020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa16e0c0 0x563baa16c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47eb58c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563baa1879e0 con 0x563baa1669f0 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47eb58c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563baa1f2460 con 0x563baa166610 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47eb58c740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x563baa1348a0 con 0x563baa16e0c0 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47d9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa172020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47d9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa172020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45176/0 (socket says 172.21.15.161:45176) 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47d9d9b640 1 -- 172.21.15.161:0/3292583645 learned_addr learned my addr 172.21.15.161:0/3292583645 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47d9d9b640 1 -- 172.21.15.161:0/3292583645 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa16e0c0 msgr2=0x563baa16c5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/3292583645 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa16e0c0 0x563baa16c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47d9d9b640 1 -- 172.21.15.161:0/3292583645 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 msgr2=0x563baa134ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/3292583645 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 0x563baa134ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.924+0000 7f47d9d9b640 1 -- 172.21.15.161:0/3292583645 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563baa040a80 con 0x563baa1669f0 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.925+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/3292583645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa172020 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f47e40b4f00 tx=0x7f47cc013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cdd17f2d32a62d9d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.925+0000 7f47c37fe640 1 -- 172.21.15.161:0/3292583645 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc00dad0 con 0x563baa1669f0 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.925+0000 7f47c37fe640 1 -- 172.21.15.161:0/3292583645 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47cc00dc30 con 0x563baa1669f0 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.925+0000 7f47c37fe640 1 -- 172.21.15.161:0/3292583645 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc026180 con 0x563baa1669f0 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 -- 172.21.15.161:0/3292583645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 msgr2=0x563baa172020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 --2- 172.21.15.161:0/3292583645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa172020 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f47e40b4f00 tx=0x7f47cc013000 comp rx=0 tx=0).stop 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 -- 172.21.15.161:0/3292583645 shutdown_connections 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 --2- 172.21.15.161:0/3292583645 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa16e0c0 0x563baa16c5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 --2- 172.21.15.161:0/3292583645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa172020 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 --2- 172.21.15.161:0/3292583645 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa166610 0x563baa134ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 -- 172.21.15.161:0/3292583645 >> 172.21.15.161:0/3292583645 conn(0x563baa139af0 msgr2=0x563baa184b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 -- 172.21.15.161:0/3292583645 shutdown_connections 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 -- 172.21.15.161:0/3292583645 wait complete. 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 -- start start 2024-03-23T14:37:03.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.926+0000 7f47eb58c740 1 Processor -- start 2024-03-23T14:37:03.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa166610 0x563baa1a68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa16c5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47eb58c740 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa16cae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47eb58c740 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x563baa1fcd90 con 0x563baa1669f0 2024-03-23T14:37:03.928 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47eb58c740 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x563baa1a5400 con 0x563baa16e0c0 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47eb58c740 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x563baa1a70d0 con 0x563baa166610 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47eb58c740 10 client.0 _ll_register_callbacks cb 0x7f47d4007f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47d9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa16c5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47d9d9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa16c5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45182/0 (socket says 172.21.15.161:45182) 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47d9d9b640 1 -- 172.21.15.161:0/4232399731 learned_addr learned my addr 172.21.15.161:0/4232399731 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa16cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.927+0000 7f47e259c640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa166610 0x563baa1a68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47d9d9b640 1 -- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa166610 msgr2=0x563baa1a68d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa166610 0x563baa1a68d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47d9d9b640 1 -- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 msgr2=0x563baa16cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa16cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47d9d9b640 1 -- 172.21.15.161:0/4232399731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47e40d0460 con 0x563baa1669f0 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47e2d9d640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa16cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:03.929 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47e259c640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa166610 0x563baa1a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47d9d9b640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa16c5a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009010 tx=0x7f47cc000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5238cc7cd5106e12 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47c17fa640 10 client.0 ms_handle_connect on v2:172.21.15.161:3300/0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47c17fa640 1 -- 172.21.15.161:0/4232399731 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47cc026380 con 0x563baa1669f0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47c17fa640 1 -- 172.21.15.161:0/4232399731 <== mon.0 v2:172.21.15.161:3300/0 2 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f47cc0264e0 con 0x563baa1669f0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 1 -- 172.21.15.161:0/4232399731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47dc030c70 con 0x563baa1669f0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47c17fa640 1 -- 172.21.15.161:0/4232399731 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f47cc0266e0 con 0x563baa1669f0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 10 client.6072 fetch_fsmap learned FSMap version 23 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 1 -- 172.21.15.161:0/4232399731 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=2+}) -- 0x563baa0667e0 con 0x563baa1669f0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47c17fa640 1 -- 172.21.15.161:0/4232399731 <== mon.0 v2:172.21.15.161:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f47dc030c70 con 0x563baa1669f0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47c17fa640 1 -- 172.21.15.161:0/4232399731 <== mon.0 v2:172.21.15.161:3300/0 5 ==== fsmap(e 23) ==== 3494+0+0 (secure 0 0 0) 0x7f47cc026d20 con 0x563baa1669f0 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 10 client.6072 fetch_fsmap finished waiting for FSMap version 23 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 10 client.6072 resolve_mds: resolved name 'a' to daemon mds.a 2024-03-23T14:37:03.930 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 20 client.6072 populate_metadata read hostname 'smithi161' 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] conn(0x563baa138cf0 0x563baa1a40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 4 client.6072 mds_command: new command op to 5925 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.928+0000 7f47eb58c740 1 -- 172.21.15.161:0/4232399731 --> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] -- command(tid 0: {"prefix": "status"}) -- 0x563ba9de2610 con 0x563baa138cf0 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47e259c640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] conn(0x563baa138cf0 0x563baa1a40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47c17fa640 10 client.6072 ms_handle_connect on v2:172.21.15.161:6833/1671955402 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47e259c640 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] conn(0x563baa138cf0 0x563baa1a40c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2346ef777f0fdf5 server_cookie=a34378a6384086a4 in_seq=0 out_seq=0 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47c17fa640 1 -- 172.21.15.161:0/4232399731 <== mds.0 v2:172.21.15.161:6833/1671955402 1 ==== command_reply(tid 0: 0 ) ==== 8+0+324 (crc 0 0 0) 0x563ba9de2610 con 0x563baa138cf0 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47c17fa640 10 client.6072 handle_command_reply: tid=0 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 client.6072 shutdown 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 -- 172.21.15.161:0/4232399731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 msgr2=0x563baa16c5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa16c5a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7f47cc009010 tx=0x7f47cc000bc0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 -- 172.21.15.161:0/4232399731 shutdown_connections 2024-03-23T14:37:03.931 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.161:6833/1671955402,v1:172.21.15.161:6835/1671955402] conn(0x563baa138cf0 0x563baa1a40c0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x563baa16e0c0 0x563baa16cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x563baa1669f0 0x563baa16c5a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47c17fa640 0 client.6072 ms_handle_reset on v2:172.21.15.161:6833/1671955402 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 --2- 172.21.15.161:0/4232399731 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x563baa166610 0x563baa1a68d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 -- 172.21.15.161:0/4232399731 >> 172.21.15.161:0/4232399731 conn(0x563baa139af0 msgr2=0x563baa13d180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 -- 172.21.15.161:0/4232399731 shutdown_connections 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 1 -- 172.21.15.161:0/4232399731 wait complete. 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.929+0000 7f47eb58c740 20 client.6072 trim_cache size 0 max 16384 2024-03-23T14:37:03.932 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "cluster_fsid": "9a16b48f-76af-4198-8fe3-d2c20a776a3e", 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "whoami": 0, 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 5925, 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "want_state": "up:active", 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "state": "up:active", 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs", 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "rank_uptime": 4.760895359, 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "mdsmap_epoch": 23, 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch": 48, 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "osdmap_epoch_barrier": 48, 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout: "uptime": 9.8104838690000005 2024-03-23T14:37:03.933 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:03.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 -- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f47c406b810 msgr2=0x7f47c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f47c406b810 0x7f47c406dcd0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f47e41607d0 tx=0x7f47cc009040 comp rx=0 tx=0).stop 2024-03-23T14:37:03.934 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 -- 172.21.15.161:0/3093384801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40cf9d0 msgr2=0x7f47e415d4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40cf9d0 0x7f47e415d4a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f47d4007e60 tx=0x7f47d400e4b0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 -- 172.21.15.161:0/3093384801 shutdown_connections 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f47c406b810 0x7f47c406dcd0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f47e415e400 0x7f47e40b5bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f47e40d2b40 0x7f47e40b5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 --2- 172.21.15.161:0/3093384801 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f47e40cf9d0 0x7f47e415d4a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 -- 172.21.15.161:0/3093384801 >> 172.21.15.161:0/3093384801 conn(0x7f47e409d190 msgr2=0x7f47e40b1cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 -- 172.21.15.161:0/3093384801 shutdown_connections 2024-03-23T14:37:03.935 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:03.930+0000 7f47e89af640 1 -- 172.21.15.161:0/3093384801 wait complete. 2024-03-23T14:37:03.937 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:37:04.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.042+0000 7fb90f78a640 1 Processor -- start 2024-03-23T14:37:04.038 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90f78a640 1 -- start start 2024-03-23T14:37:04.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90f78a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb90815e400 0x7fb9080cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:04.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90f78a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb9080cf9d0 0x7fb9080d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:04.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90f78a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb9080d0460 con 0x7fb9080d1470 2024-03-23T14:37:04.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90f78a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb9080d1050 con 0x7fb90815e400 2024-03-23T14:37:04.039 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90f78a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fb9080d06b0 con 0x7fb9080cf9d0 2024-03-23T14:37:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90d526640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb90815e400 0x7fb9080cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90dd27640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb9080d1470 0x7fb9080b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42896/0 (socket says 172.21.15.161:42896) 2024-03-23T14:37:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90d526640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb90815e400 0x7fb9080cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45188/0 (socket says 172.21.15.161:45188) 2024-03-23T14:37:04.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.043+0000 7fb90dd27640 1 -- 172.21.15.161:0/1513639170 learned_addr learned my addr 172.21.15.161:0/1513639170 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.044+0000 7fb90d526640 1 -- 172.21.15.161:0/1513639170 >> v2:172.21.15.183:3300/0 conn(0x7fb9080cf9d0 msgr2=0x7fb9080d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.044+0000 7fb90cd25640 1 --2- 172.21.15.161:0/1513639170 >> v2:172.21.15.183:3300/0 conn(0x7fb9080cf9d0 0x7fb9080d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.044+0000 7fb8fffff640 1 -- 172.21.15.161:0/1513639170 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1752960284 0 0) 0x7fb9080d0460 con 0x7fb9080d1470 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.044+0000 7fb90d526640 1 --2- 172.21.15.161:0/1513639170 >> v2:172.21.15.183:3300/0 conn(0x7fb9080cf9d0 0x7fb9080d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.044+0000 7fb90d526640 1 -- 172.21.15.161:0/1513639170 >> v1:172.21.15.161:6789/0 conn(0x7fb9080d1470 legacy=0x7fb9080b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.044+0000 7fb90d526640 1 -- 172.21.15.161:0/1513639170 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9080d0990 con 0x7fb90815e400 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.044+0000 7fb90cd25640 1 --2- 172.21.15.161:0/1513639170 >> v2:172.21.15.183:3300/0 conn(0x7fb9080cf9d0 0x7fb9080d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb90d526640 1 --2- 172.21.15.161:0/1513639170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90815e400 0x7fb9080cf490 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb9080b6a20 tx=0x7fb90402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=183b563fde4a0902 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb8fffff640 1 -- 172.21.15.161:0/1513639170 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904031830 con 0x7fb90815e400 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb8fffff640 1 -- 172.21.15.161:0/1513639170 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb904031990 con 0x7fb90815e400 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb8fffff640 1 -- 172.21.15.161:0/1513639170 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb904031d20 con 0x7fb90815e400 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb90f78a640 1 -- 172.21.15.161:0/1513639170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90815e400 msgr2=0x7fb9080cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb90f78a640 1 --2- 172.21.15.161:0/1513639170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90815e400 0x7fb9080cf490 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fb9080b6a20 tx=0x7fb90402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb90f78a640 1 -- 172.21.15.161:0/1513639170 shutdown_connections 2024-03-23T14:37:04.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb90f78a640 1 --2- 172.21.15.161:0/1513639170 >> v2:172.21.15.183:3300/0 conn(0x7fb9080cf9d0 0x7fb9080d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb90f78a640 1 --2- 172.21.15.161:0/1513639170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb90815e400 0x7fb9080cf490 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.045+0000 7fb90f78a640 1 -- 172.21.15.161:0/1513639170 >> 172.21.15.161:0/1513639170 conn(0x7fb90809d190 msgr2=0x7fb9080d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:04.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.046+0000 7fb90f78a640 1 -- 172.21.15.161:0/1513639170 shutdown_connections 2024-03-23T14:37:04.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.046+0000 7fb90f78a640 1 -- 172.21.15.161:0/1513639170 wait complete. 2024-03-23T14:37:04.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.047+0000 7fb90f78a640 1 Processor -- start 2024-03-23T14:37:04.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.047+0000 7fb90f78a640 1 -- start start 2024-03-23T14:37:04.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90f78a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9080cf9d0 0x7fb908160200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90f78a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9080d1470 0x7fb9080b6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90f78a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb90815e400 0x7fb9080c9da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90f78a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb908160740 con 0x7fb9080cf9d0 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90f78a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb9080c7980 con 0x7fb9080d1470 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90f78a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb9080b7120 con 0x7fb90815e400 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90d526640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9080cf9d0 0x7fb908160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90d526640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9080cf9d0 0x7fb908160200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45200/0 (socket says 172.21.15.161:45200) 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90d526640 1 -- 172.21.15.161:0/432161206 learned_addr learned my addr 172.21.15.161:0/432161206 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90cd25640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9080d1470 0x7fb9080b6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.048+0000 7fb90dd27640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb90815e400 0x7fb9080c9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90d526640 1 -- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb90815e400 msgr2=0x7fb9080c9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90d526640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb90815e400 0x7fb9080c9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90d526640 1 -- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9080d1470 msgr2=0x7fb9080b6be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:04.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90d526640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9080d1470 0x7fb9080b6be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90d526640 1 -- 172.21.15.161:0/432161206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb904009400 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90dd27640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb90815e400 0x7fb9080c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90d526640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9080cf9d0 0x7fb908160200 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb904000c00 tx=0x7fb9040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb8fdffb640 1 -- 172.21.15.161:0/432161206 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90403aa60 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb8fdffb640 1 -- 172.21.15.161:0/432161206 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb90403abe0 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9080c7c20 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb8fdffb640 1 -- 172.21.15.161:0/432161206 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9040443f0 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.049+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9080c81e0 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.050+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9080a1e00 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.051+0000 7fb8fdffb640 1 -- 172.21.15.161:0/432161206 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb90404d450 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.051+0000 7fb8fdffb640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8ec06b810 0x7fb8ec06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.051+0000 7fb8fdffb640 1 -- 172.21.15.161:0/432161206 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fb9040ac420 con 0x7fb9080cf9d0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.051+0000 7fb90cd25640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8ec06b810 0x7fb8ec06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:04.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.052+0000 7fb90cd25640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8ec06b810 0x7fb8ec06dcd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb90806d320 tx=0x7fb8f8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:04.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.055+0000 7fb8fdffb640 1 -- 172.21.15.161:0/432161206 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb904076e40 con 0x7fb9080cf9d0 2024-03-23T14:37:04.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.269+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fb9080c9b80 con 0x7fb9080cf9d0 2024-03-23T14:37:04.265 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.269+0000 7fb8fdffb640 1 -- 172.21.15.161:0/432161206 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7fb90407a500 con 0x7fb9080cf9d0 2024-03-23T14:37:04.265 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T14:37:04.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8ec06b810 msgr2=0x7fb8ec06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8ec06b810 0x7fb8ec06dcd0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb90806d320 tx=0x7fb8f8008040 comp rx=0 tx=0).stop 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9080cf9d0 msgr2=0x7fb908160200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9080cf9d0 0x7fb908160200 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fb904000c00 tx=0x7fb9040046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 shutdown_connections 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb8ec06b810 0x7fb8ec06dcd0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb90815e400 0x7fb9080c9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb9080d1470 0x7fb9080b6be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 --2- 172.21.15.161:0/432161206 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb9080cf9d0 0x7fb908160200 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 >> 172.21.15.161:0/432161206 conn(0x7fb90809d190 msgr2=0x7fb9080b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 shutdown_connections 2024-03-23T14:37:04.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04.271+0000 7fb90f78a640 1 -- 172.21.15.161:0/432161206 wait complete. 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:37:04.278 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:04.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:04.279 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-23T14:37:04.294 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42646sec preferred_lft 42646sec 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T14:37:04.295 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c4a9:64ff:fe59:877a/64 scope link 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c48:50ff:fe01:9f4e/64 scope link 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 62:02:d5:17:e7:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::6002:d5ff:fe17:e70d/64 scope link 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:04.296 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:04.297 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:37:04.361 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-23T14:37:04.362 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:04.363 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-23T14:37:04.418 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-23T14:37:04.418 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:37:04.418 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:37:04.418 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:37:04.418 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:37:04.418 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:37:04.418 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:37:04.418 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:37:04.419 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:37:04.419 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:37:04.419 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:37:04.435 INFO:tasks.ceph.mds.a.smithi161.stderr:2024-03-23T14:37:04.439+0000 7f76d14c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T14:37:04.482 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:37:04.525 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:37:04.531 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:04.531 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:37:04.546 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T14:37:04.546 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T14:37:04.547 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:04.547 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:04.547 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-23T14:37:04.613 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T14:37:04.614 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:37:04.614 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:04.668 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:04.668 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:37:04.668 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:04.724 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:37:04.792 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:37:04.848 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:37:04.848 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:37:04.848 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:37:04.848 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:37:04.849 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:37:04.850 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:37:04.851 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:37:04.851 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:37:04.851 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:04.851 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:04.851 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:04.851 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:04.851 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:04.851 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:37:04.903 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T14:37:04.903 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T14:37:04.946 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:37:04.973 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:37:05.029 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:37:05.029 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:37:05.030 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:37:05.031 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:37:05.032 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:37:05.032 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:37:05.032 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:05.032 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:05.032 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:05.032 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:05.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:05.032 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:37:05.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:2024-03-23T14:37:05.049+0000 7f6213f77480 -1 init, newargv = 0x557f16291940 newargc=15 2024-03-23T14:37:05.045 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[56996]: starting ceph client 2024-03-23T14:37:05.049 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[56996]: starting fuse 2024-03-23T14:37:05.059 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:37:05.059 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T14:37:05.060 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-23T14:37:05.061 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:37:05.062 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:05.062 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.0")) 2024-03-23T14:37:05.062 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:37:05.182 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.0.56996.asok 2024-03-23T14:37:05.186 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:37:05.193 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.56996.asok 2024-03-23T14:37:05.194 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:05.194 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.56996.asok status 2024-03-23T14:37:05.348 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:05.348 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "0", 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "56996", 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6087, 2024-03-23T14:37:05.349 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 6087 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3894765325 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:37:05.350 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3894765325 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.6087 192.168.0.1:0/3894765325", 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/3894765325", 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 23, 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 48, 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-23T14:37:05.351 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:05.357 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:05.357 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-23T14:37:05.357 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:05.373 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-23T14:37:05.373 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:05.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:05.373 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:05.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:05.441 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest/mnt.0 && exec touch test_file) 2024-03-23T14:37:05.502 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow r pool=cephfs_data' 2024-03-23T14:37:05.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.653+0000 7f7bd904e640 1 Processor -- start 2024-03-23T14:37:05.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.653+0000 7f7bd904e640 1 -- start start 2024-03-23T14:37:05.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd904e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7bd40d2bf0 0x7f7bd40d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd904e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d1d50 0x7f7bd40b46b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd904e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7bd40b0ac0 con 0x7f7bd415e440 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd904e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f7bd40b4220 con 0x7f7bd40d2bf0 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd904e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd40b3350 con 0x7f7bd40d1d50 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd359e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d1d50 0x7f7bd40b46b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd259c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7bd40d2bf0 0x7f7bd40d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd359e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d1d50 0x7f7bd40b46b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:47330/0 (socket says 172.21.15.161:47330) 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd259c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7bd40d2bf0 0x7f7bd40d4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:44310/0 (socket says 172.21.15.161:44310) 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd359e640 1 -- 172.21.15.161:0/3235532140 learned_addr learned my addr 172.21.15.161:0/3235532140 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:05.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.654+0000 7f7bd2d9d640 1 --1- 172.21.15.161:0/3235532140 >> v1:172.21.15.183:6789/0 conn(0x7f7bd415e440 0x7f7bd40d24e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:46466/0 (socket says 172.21.15.161:46466) 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.655+0000 7f7bd359e640 1 -- 172.21.15.161:0/3235532140 >> v2:172.21.15.183:3300/0 conn(0x7f7bd40d2bf0 msgr2=0x7f7bd40d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.655+0000 7f7bd359e640 1 --2- 172.21.15.161:0/3235532140 >> v2:172.21.15.183:3300/0 conn(0x7f7bd40d2bf0 0x7f7bd40d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.655+0000 7f7bd359e640 1 -- 172.21.15.161:0/3235532140 >> v1:172.21.15.183:6789/0 conn(0x7f7bd415e440 legacy=0x7f7bd40d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.655+0000 7f7bd359e640 1 -- 172.21.15.161:0/3235532140 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bd40b3610 con 0x7f7bd40d1d50 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.655+0000 7f7bd259c640 1 --2- 172.21.15.161:0/3235532140 >> v2:172.21.15.183:3300/0 conn(0x7f7bd40d2bf0 0x7f7bd40d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd359e640 1 --2- 172.21.15.161:0/3235532140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d1d50 0x7f7bd40b46b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009e50 tx=0x7f7bcc02bf60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae210ff540cba1e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd1d9b640 1 -- 172.21.15.161:0/3235532140 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc02cb00 con 0x7f7bd40d1d50 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd1d9b640 1 -- 172.21.15.161:0/3235532140 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bcc02cc60 con 0x7f7bd40d1d50 2024-03-23T14:37:05.653 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd1d9b640 1 -- 172.21.15.161:0/3235532140 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc0343f0 con 0x7f7bd40d1d50 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd904e640 1 -- 172.21.15.161:0/3235532140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d1d50 msgr2=0x7f7bd40b46b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd904e640 1 --2- 172.21.15.161:0/3235532140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d1d50 0x7f7bd40b46b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009e50 tx=0x7f7bcc02bf60 comp rx=0 tx=0).stop 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd904e640 1 -- 172.21.15.161:0/3235532140 shutdown_connections 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd904e640 1 --2- 172.21.15.161:0/3235532140 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d1d50 0x7f7bd40b46b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd904e640 1 --2- 172.21.15.161:0/3235532140 >> v2:172.21.15.183:3300/0 conn(0x7f7bd40d2bf0 0x7f7bd40d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd904e640 1 -- 172.21.15.161:0/3235532140 >> 172.21.15.161:0/3235532140 conn(0x7f7bd40d3f00 msgr2=0x7f7bd40d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd904e640 1 -- 172.21.15.161:0/3235532140 shutdown_connections 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.656+0000 7f7bd904e640 1 -- 172.21.15.161:0/3235532140 wait complete. 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.658+0000 7f7bd904e640 1 Processor -- start 2024-03-23T14:37:05.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.658+0000 7f7bd904e640 1 -- start start 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd904e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bd40d1d50 0x7f7bd41f4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd904e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d2bf0 0x7f7bd40cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd904e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd415e440 0x7f7bd40cd430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd904e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd40cc650 con 0x7f7bd415e440 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd904e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7bd40b9bf0 con 0x7f7bd40d1d50 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd904e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7bd40cb070 con 0x7f7bd40d2bf0 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd415e440 0x7f7bd40cd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd415e440 0x7f7bd40cd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45202/0 (socket says 172.21.15.161:45202) 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 -- 172.21.15.161:0/1252956684 learned_addr learned my addr 172.21.15.161:0/1252956684 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd2d9d640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bd40d1d50 0x7f7bd41f4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd259c640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d2bf0 0x7f7bd40cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 -- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d2bf0 msgr2=0x7f7bd40cc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d2bf0 0x7f7bd40cc110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 -- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bd40d1d50 msgr2=0x7f7bd41f4c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bd40d1d50 0x7f7bd41f4c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 -- 172.21.15.161:0/1252956684 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7bc8009400 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd359e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd415e440 0x7f7bd40cd430 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009e50 tx=0x7f7bcc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/1252956684 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc009510 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/1252956684 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7bcc03e450 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7bcc009b30 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/1252956684 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bcc03e800 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.659+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7bd40cb630 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.660+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7bd40b46b0 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.661+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/1252956684 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7bcc03e960 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.661+0000 7f7bbb7fe640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7bc006b840 0x7f7bc006dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.661+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/1252956684 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f7bcc0aca60 con 0x7f7bd415e440 2024-03-23T14:37:05.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.661+0000 7f7bd2d9d640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7bc006b840 0x7f7bc006dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:05.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.662+0000 7f7bd2d9d640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7bc006b840 0x7f7bc006dd00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4007fd0 tx=0x7f7bc4007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:05.665 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.670+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/1252956684 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7bcc077530 con 0x7f7bd415e440 2024-03-23T14:37:05.913 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.917+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]} v 0) -- 0x7f7bd4068160 con 0x7f7bd415e440 2024-03-23T14:37:05.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.920+0000 7f7bbb7fe640 1 -- 172.21.15.161:0/1252956684 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow r pool=cephfs_data"]}]=0 updated caps for client.0 v16) ==== 183+0+140 (secure 0 0 0) 0x7f7bcc07abf0 con 0x7f7bd415e440 2024-03-23T14:37:05.915 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:37:05.916 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:37:05.916 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-23T14:37:05.916 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-23T14:37:05.916 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow r pool=cephfs_data" 2024-03-23T14:37:05.916 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-23T14:37:05.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7bc006b840 msgr2=0x7f7bc006dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:05.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7bc006b840 0x7f7bc006dd00 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f7bc4007fd0 tx=0x7f7bc4007e80 comp rx=0 tx=0).stop 2024-03-23T14:37:05.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd415e440 msgr2=0x7f7bd40cd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:05.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd415e440 0x7f7bd40cd430 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f7bcc009e50 tx=0x7f7bcc004a40 comp rx=0 tx=0).stop 2024-03-23T14:37:05.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 shutdown_connections 2024-03-23T14:37:05.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7bc006b840 0x7f7bc006dd00 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.919 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7bd415e440 0x7f7bd40cd430 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7bd40d2bf0 0x7f7bd40cc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 --2- 172.21.15.161:0/1252956684 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7bd40d1d50 0x7f7bd41f4c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:05.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 >> 172.21.15.161:0/1252956684 conn(0x7f7bd40d3f00 msgr2=0x7f7bd40cf2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:05.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 shutdown_connections 2024-03-23T14:37:05.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:05.923+0000 7f7bd904e640 1 -- 172.21.15.161:0/1252956684 wait complete. 2024-03-23T14:37:05.930 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:37:05.930 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:05.945 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:06.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.090+0000 7f42d6ced640 1 Processor -- start 2024-03-23T14:37:06.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.090+0000 7f42d6ced640 1 -- start start 2024-03-23T14:37:06.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42d6ced640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42d015e400 0x7f42d00cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42d6ced640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f42d00d2b40 0x7f42d00b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42d6ced640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42d00d0460 con 0x7f42d00cf9d0 2024-03-23T14:37:06.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42d6ced640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f42d00d1050 con 0x7f42d015e400 2024-03-23T14:37:06.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42d6ced640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f42d00d06b0 con 0x7f42d00d2b40 2024-03-23T14:37:06.088 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42d4a89640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42d015e400 0x7f42d00cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42cffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f42d00cf9d0 0x7f42d00d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42910/0 (socket says 172.21.15.161:42910) 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42cffff640 1 -- 172.21.15.161:0/1371193155 learned_addr learned my addr 172.21.15.161:0/1371193155 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.091+0000 7f42d4a89640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f42d015e400 0x7f42d00cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45216/0 (socket says 172.21.15.161:45216) 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.092+0000 7f42d528a640 1 --2- 172.21.15.161:0/1371193155 >> v2:172.21.15.183:3300/0 conn(0x7f42d00d2b40 0x7f42d00b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.092+0000 7f42d4a89640 1 -- 172.21.15.161:0/1371193155 >> v2:172.21.15.183:3300/0 conn(0x7f42d00d2b40 msgr2=0x7f42d00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.092+0000 7f42d4a89640 1 --2- 172.21.15.161:0/1371193155 >> v2:172.21.15.183:3300/0 conn(0x7f42d00d2b40 0x7f42d00b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.092+0000 7f42d4a89640 1 -- 172.21.15.161:0/1371193155 >> v1:172.21.15.161:6789/0 conn(0x7f42d00cf9d0 legacy=0x7f42d00d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.092+0000 7f42d4a89640 1 -- 172.21.15.161:0/1371193155 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42d00d0990 con 0x7f42d015e400 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.092+0000 7f42d528a640 1 --2- 172.21.15.161:0/1371193155 >> v2:172.21.15.183:3300/0 conn(0x7f42d00d2b40 0x7f42d00b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:06.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d4a89640 1 --2- 172.21.15.161:0/1371193155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d015e400 0x7f42d00cf490 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f42d00ce430 tx=0x7f42c402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b738d21d09ff21c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42cf7fe640 1 -- 172.21.15.161:0/1371193155 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4031830 con 0x7f42d015e400 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42cf7fe640 1 -- 172.21.15.161:0/1371193155 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42c4031990 con 0x7f42d015e400 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42cf7fe640 1 -- 172.21.15.161:0/1371193155 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c4031d20 con 0x7f42d015e400 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d6ced640 1 -- 172.21.15.161:0/1371193155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d015e400 msgr2=0x7f42d00cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d6ced640 1 --2- 172.21.15.161:0/1371193155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d015e400 0x7f42d00cf490 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f42d00ce430 tx=0x7f42c402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d6ced640 1 -- 172.21.15.161:0/1371193155 shutdown_connections 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d6ced640 1 --2- 172.21.15.161:0/1371193155 >> v2:172.21.15.183:3300/0 conn(0x7f42d00d2b40 0x7f42d00b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d6ced640 1 --2- 172.21.15.161:0/1371193155 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d015e400 0x7f42d00cf490 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d6ced640 1 -- 172.21.15.161:0/1371193155 >> 172.21.15.161:0/1371193155 conn(0x7f42d009d190 msgr2=0x7f42d00d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d6ced640 1 -- 172.21.15.161:0/1371193155 shutdown_connections 2024-03-23T14:37:06.093 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.093+0000 7f42d6ced640 1 -- 172.21.15.161:0/1371193155 wait complete. 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.095+0000 7f42d6ced640 1 Processor -- start 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.095+0000 7f42d6ced640 1 -- start start 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.095+0000 7f42d6ced640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d00cf9d0 0x7f42d0158ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.095+0000 7f42d6ced640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42d00d2b40 0x7f42d015b210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.095+0000 7f42d6ced640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42d015e400 0x7f42d015b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.095+0000 7f42d6ced640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42d00ccfa0 con 0x7f42d00cf9d0 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.095+0000 7f42d6ced640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f42d0159430 con 0x7f42d00d2b40 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.095+0000 7f42d6ced640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f42d00cc9e0 con 0x7f42d015e400 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d00cf9d0 0x7f42d0158ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d00cf9d0 0x7f42d0158ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45222/0 (socket says 172.21.15.161:45222) 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 -- 172.21.15.161:0/3024909333 learned_addr learned my addr 172.21.15.161:0/3024909333 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:06.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42cffff640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42d00d2b40 0x7f42d015b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d528a640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42d015e400 0x7f42d015b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 -- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42d015e400 msgr2=0x7f42d015b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42d015e400 0x7f42d015b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 -- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42d00d2b40 msgr2=0x7f42d015b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42d00d2b40 0x7f42d015b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 -- 172.21.15.161:0/3024909333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c4009400 con 0x7f42d00cf9d0 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d4a89640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d00cf9d0 0x7f42d0158ef0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f42c4000c00 tx=0x7f42c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42cd7fa640 1 -- 172.21.15.161:0/3024909333 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c403aa60 con 0x7f42d00cf9d0 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42cd7fa640 1 -- 172.21.15.161:0/3024909333 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f42c403abe0 con 0x7f42d00cf9d0 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42d00cccc0 con 0x7f42d00cf9d0 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42cd7fa640 1 -- 172.21.15.161:0/3024909333 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42c40443f0 con 0x7f42d00cf9d0 2024-03-23T14:37:06.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.096+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42d015f220 con 0x7f42d00cf9d0 2024-03-23T14:37:06.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.097+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42d015eda0 con 0x7f42d00cf9d0 2024-03-23T14:37:06.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.098+0000 7f42cd7fa640 1 -- 172.21.15.161:0/3024909333 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f42c404d450 con 0x7f42d00cf9d0 2024-03-23T14:37:06.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.099+0000 7f42cd7fa640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42b406b810 0x7f42b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.099+0000 7f42cd7fa640 1 -- 172.21.15.161:0/3024909333 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f42c40ac460 con 0x7f42d00cf9d0 2024-03-23T14:37:06.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.099+0000 7f42cffff640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42b406b810 0x7f42b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.099+0000 7f42cffff640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42b406b810 0x7f42b406dcd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f42d00cae30 tx=0x7f42bc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.103+0000 7f42cd7fa640 1 -- 172.21.15.161:0/3024909333 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f42d015eda0 con 0x7f42d00cf9d0 2024-03-23T14:37:06.278 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.283+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f42d015eda0 con 0x7f42d00cf9d0 2024-03-23T14:37:06.279 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.284+0000 7f42cd7fa640 1 -- 172.21.15.161:0/3024909333 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7f42d015eda0 con 0x7f42d00cf9d0 2024-03-23T14:37:06.280 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:06.280 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42b406b810 msgr2=0x7f42b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42b406b810 0x7f42b406dcd0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f42d00cae30 tx=0x7f42bc009040 comp rx=0 tx=0).stop 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d00cf9d0 msgr2=0x7f42d0158ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d00cf9d0 0x7f42d0158ef0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f42c4000c00 tx=0x7f42c40046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 shutdown_connections 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f42b406b810 0x7f42b406dcd0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f42d015e400 0x7f42d015b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f42d00d2b40 0x7f42d015b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.282 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 --2- 172.21.15.161:0/3024909333 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42d00cf9d0 0x7f42d0158ef0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 >> 172.21.15.161:0/3024909333 conn(0x7f42d009d190 msgr2=0x7f42d00b1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:06.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 shutdown_connections 2024-03-23T14:37:06.283 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.286+0000 7f42d6ced640 1 -- 172.21.15.161:0/3024909333 wait complete. 2024-03-23T14:37:06.293 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:56.747667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:57.478793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:37:06.293 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:37:06.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.402+0000 7f20f7b1e640 1 Processor -- start 2024-03-23T14:37:06.398 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.402+0000 7f20f7b1e640 1 -- start start 2024-03-23T14:37:06.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f7b1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f0161820 0x7f20f00d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f7b1e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f20f00d1270 0x7f20f00d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f7b1e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20f0161c00 con 0x7f20f00d29d0 2024-03-23T14:37:06.399 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f7b1e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f20f00d2e20 con 0x7f20f00d1270 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f7b1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f20f00cea30 con 0x7f20f0161820 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f50b9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f20f00d1270 0x7f20f00d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f50b9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f20f00d1270 0x7f20f00d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45228/0 (socket says 172.21.15.161:45228) 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f50b9640 1 -- 172.21.15.161:0/3066658376 learned_addr learned my addr 172.21.15.161:0/3066658376 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f58ba640 1 --2- 172.21.15.161:0/3066658376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f0161820 0x7f20f00d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f50b9640 1 -- 172.21.15.161:0/3066658376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f0161820 msgr2=0x7f20f00d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f50b9640 1 --2- 172.21.15.161:0/3066658376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f0161820 0x7f20f00d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f50b9640 1 -- 172.21.15.161:0/3066658376 >> v1:172.21.15.183:6789/0 conn(0x7f20f00d29d0 legacy=0x7f20f0161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f50b9640 1 -- 172.21.15.161:0/3066658376 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20f00cfd70 con 0x7f20f00d1270 2024-03-23T14:37:06.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.403+0000 7f20f58ba640 1 --2- 172.21.15.161:0/3066658376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f0161820 0x7f20f00d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:06.401 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.404+0000 7f20f50b9640 1 --2- 172.21.15.161:0/3066658376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f00d2490 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009870 tx=0x7f20e402b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=62fdbb08f40f5963 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.404+0000 7f20f48b8640 1 -- 172.21.15.161:0/3066658376 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e4031640 con 0x7f20f00d1270 2024-03-23T14:37:06.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.404+0000 7f20f48b8640 1 -- 172.21.15.161:0/3066658376 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e40317a0 con 0x7f20f00d1270 2024-03-23T14:37:06.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.404+0000 7f20f48b8640 1 -- 172.21.15.161:0/3066658376 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e4031b30 con 0x7f20f00d1270 2024-03-23T14:37:06.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.405+0000 7f20f7b1e640 1 -- 172.21.15.161:0/3066658376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 msgr2=0x7f20f00d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.405+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/3066658376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f00d2490 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f20e4009870 tx=0x7f20e402b8f0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.405+0000 7f20f7b1e640 1 -- 172.21.15.161:0/3066658376 shutdown_connections 2024-03-23T14:37:06.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.405+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/3066658376 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f00d2490 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.402 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.405+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/3066658376 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f0161820 0x7f20f00d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.405+0000 7f20f7b1e640 1 -- 172.21.15.161:0/3066658376 >> 172.21.15.161:0/3066658376 conn(0x7f20f009d190 msgr2=0x7f20f00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:06.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.405+0000 7f20f7b1e640 1 -- 172.21.15.161:0/3066658376 shutdown_connections 2024-03-23T14:37:06.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.405+0000 7f20f7b1e640 1 -- 172.21.15.161:0/3066658376 wait complete. 2024-03-23T14:37:06.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.406+0000 7f20f7b1e640 1 Processor -- start 2024-03-23T14:37:06.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.406+0000 7f20f7b1e640 1 -- start start 2024-03-23T14:37:06.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f7b1e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f0158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.405 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f7b1e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f00d29d0 0x7f20f00c9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f7b1e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20f0161820 0x7f20f00cb360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f7b1e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f20f00c9940 con 0x7f20f00d1270 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f7b1e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f20f00cb8a0 con 0x7f20f0161820 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f7b1e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f20f015ee10 con 0x7f20f00d29d0 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f58ba640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f0158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f58ba640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f0158d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45238/0 (socket says 172.21.15.161:45238) 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f58ba640 1 -- 172.21.15.161:0/2083789495 learned_addr learned my addr 172.21.15.161:0/2083789495 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f58ba640 1 -- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f00d29d0 msgr2=0x7f20f00c9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f58ba640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f00d29d0 0x7f20f00c9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20f58ba640 1 -- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20f0161820 msgr2=0x7f20f00cb360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.407+0000 7f20f60bb640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20f0161820 0x7f20f00cb360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20f58ba640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20f0161820 0x7f20f00cb360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20f58ba640 1 -- 172.21.15.161:0/2083789495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e40094d0 con 0x7f20f00d1270 2024-03-23T14:37:06.406 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20f60bb640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20f0161820 0x7f20f00cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20f58ba640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f0158d20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f20e000c7f0 tx=0x7f20e000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20ee7fc640 1 -- 172.21.15.161:0/2083789495 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0007a30 con 0x7f20f00d1270 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20ee7fc640 1 -- 172.21.15.161:0/2083789495 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e0007b90 con 0x7f20f00d1270 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20f015f0f0 con 0x7f20f00d1270 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20ee7fc640 1 -- 172.21.15.161:0/2083789495 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e000ce40 con 0x7f20f00d1270 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.408+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20f015f6b0 con 0x7f20f00d1270 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.409+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20f0068b90 con 0x7f20f00d1270 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.409+0000 7f20ee7fc640 1 -- 172.21.15.161:0/2083789495 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f20e00113f0 con 0x7f20f00d1270 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.410+0000 7f20ee7fc640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20dc06b810 0x7f20dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.410+0000 7f20ee7fc640 1 -- 172.21.15.161:0/2083789495 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f20e008a380 con 0x7f20f00d1270 2024-03-23T14:37:06.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.410+0000 7f20f50b9640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20dc06b810 0x7f20dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.410+0000 7f20f50b9640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20dc06b810 0x7f20dc06dcd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f20e4036000 tx=0x7f20e4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.409 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.414+0000 7f20ee7fc640 1 -- 172.21.15.161:0/2083789495 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f20e0054da0 con 0x7f20f00d1270 2024-03-23T14:37:06.586 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.591+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f20f004f4e0 con 0x7f20f00d1270 2024-03-23T14:37:06.587 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.591+0000 7f20ee7fc640 1 -- 172.21.15.161:0/2083789495 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f20e0058460 con 0x7f20f00d1270 2024-03-23T14:37:06.587 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:37:06.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20dc06b810 msgr2=0x7f20dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20dc06b810 0x7f20dc06dcd0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f20e4036000 tx=0x7f20e4030000 comp rx=0 tx=0).stop 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 msgr2=0x7f20f0158d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f0158d20 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f20e000c7f0 tx=0x7f20e000cbf0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 shutdown_connections 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f20dc06b810 0x7f20dc06dcd0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f20f0161820 0x7f20f00cb360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f20f00d29d0 0x7f20f00c9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 --2- 172.21.15.161:0/2083789495 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f20f00d1270 0x7f20f0158d20 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 >> 172.21.15.161:0/2083789495 conn(0x7f20f009d190 msgr2=0x7f20f00cff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 shutdown_connections 2024-03-23T14:37:06.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.593+0000 7f20f7b1e640 1 -- 172.21.15.161:0/2083789495 wait complete. 2024-03-23T14:37:06.600 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:37:06.706 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.709+0000 7f5a9bde9640 1 Processor -- start 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.709+0000 7f5a9bde9640 1 -- start start 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.710+0000 7f5a9bde9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5a940d2cd0 0x7f5a940ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.710+0000 7f5a9bde9640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a940d1cb0 0x7f5a940cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.710+0000 7f5a9bde9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a940d1330 con 0x7f5a94161750 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.710+0000 7f5a9bde9640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5a940d0720 con 0x7f5a940d1cb0 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.710+0000 7f5a9bde9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f5a940d0a00 con 0x7f5a940d2cd0 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.710+0000 7f5a9a386640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a940d1cb0 0x7f5a940cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.710+0000 7f5a9a386640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a940d1cb0 0x7f5a940cf050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45252/0 (socket says 172.21.15.161:45252) 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.710+0000 7f5a9a386640 1 -- 172.21.15.161:0/256120743 learned_addr learned my addr 172.21.15.161:0/256120743 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.711+0000 7f5a99384640 1 --2- 172.21.15.161:0/256120743 >> v2:172.21.15.183:3300/0 conn(0x7f5a940d2cd0 0x7f5a940ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.711+0000 7f5a9a386640 1 -- 172.21.15.161:0/256120743 >> v2:172.21.15.183:3300/0 conn(0x7f5a940d2cd0 msgr2=0x7f5a940ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:06.707 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.711+0000 7f5a9a386640 1 --2- 172.21.15.161:0/256120743 >> v2:172.21.15.183:3300/0 conn(0x7f5a940d2cd0 0x7f5a940ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.711+0000 7f5a9a386640 1 -- 172.21.15.161:0/256120743 >> v1:172.21.15.183:6789/0 conn(0x7f5a94161750 legacy=0x7f5a940d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:06.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.711+0000 7f5a9a386640 1 -- 172.21.15.161:0/256120743 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a940d0ce0 con 0x7f5a940d1cb0 2024-03-23T14:37:06.708 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.711+0000 7f5a99384640 1 --2- 172.21.15.161:0/256120743 >> v2:172.21.15.183:3300/0 conn(0x7f5a940d2cd0 0x7f5a940ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.711+0000 7f5a9a386640 1 --2- 172.21.15.161:0/256120743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a940cf050 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5a940c7be0 tx=0x7f5a8c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=140c57391c33b7b1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a98b83640 1 -- 172.21.15.161:0/256120743 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c02ca80 con 0x7f5a940d1cb0 2024-03-23T14:37:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a98b83640 1 -- 172.21.15.161:0/256120743 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a8c02cbe0 con 0x7f5a940d1cb0 2024-03-23T14:37:06.709 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a98b83640 1 -- 172.21.15.161:0/256120743 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a8c0343f0 con 0x7f5a940d1cb0 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a9bde9640 1 -- 172.21.15.161:0/256120743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 msgr2=0x7f5a940cf050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/256120743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a940cf050 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f5a940c7be0 tx=0x7f5a8c02bee0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a9bde9640 1 -- 172.21.15.161:0/256120743 shutdown_connections 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/256120743 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a940cf050 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/256120743 >> v2:172.21.15.183:3300/0 conn(0x7f5a940d2cd0 0x7f5a940ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a9bde9640 1 -- 172.21.15.161:0/256120743 >> 172.21.15.161:0/256120743 conn(0x7f5a9409d450 msgr2=0x7f5a940d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a9bde9640 1 -- 172.21.15.161:0/256120743 shutdown_connections 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.712+0000 7f5a9bde9640 1 -- 172.21.15.161:0/256120743 wait complete. 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.714+0000 7f5a9bde9640 1 Processor -- start 2024-03-23T14:37:06.710 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.714+0000 7f5a9bde9640 1 -- start start 2024-03-23T14:37:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9bde9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a9415dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9bde9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a940d2cd0 0x7f5a9415a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9bde9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a94161750 0x7f5a940cc2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9bde9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9415c3c0 con 0x7f5a940d1cb0 2024-03-23T14:37:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9bde9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5a9415a840 con 0x7f5a940d2cd0 2024-03-23T14:37:06.712 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9bde9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5a940cc820 con 0x7f5a94161750 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a9415dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a9415dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45266/0 (socket says 172.21.15.161:45266) 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 -- 172.21.15.161:0/782521151 learned_addr learned my addr 172.21.15.161:0/782521151 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99384640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a940d2cd0 0x7f5a9415a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9a386640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a94161750 0x7f5a940cc2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 -- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a94161750 msgr2=0x7f5a940cc2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a94161750 0x7f5a940cc2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 -- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a940d2cd0 msgr2=0x7f5a9415a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a940d2cd0 0x7f5a9415a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 -- 172.21.15.161:0/782521151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8c009b30 con 0x7f5a940d1cb0 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9a386640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a94161750 0x7f5a940cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a99b85640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a9415dcc0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000b3f0 tx=0x7f5a9000b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a8a7fc640 1 -- 172.21.15.161:0/782521151 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90011830 con 0x7f5a940d1cb0 2024-03-23T14:37:06.713 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a8a7fc640 1 -- 172.21.15.161:0/782521151 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a90011990 con 0x7f5a940d1cb0 2024-03-23T14:37:06.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a9415e3a0 con 0x7f5a940d1cb0 2024-03-23T14:37:06.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a8a7fc640 1 -- 172.21.15.161:0/782521151 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a90011d40 con 0x7f5a940d1cb0 2024-03-23T14:37:06.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.715+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a9415bfe0 con 0x7f5a940d1cb0 2024-03-23T14:37:06.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.716+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a940a2030 con 0x7f5a940d1cb0 2024-03-23T14:37:06.714 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.717+0000 7f5a8a7fc640 1 -- 172.21.15.161:0/782521151 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5a9001a400 con 0x7f5a940d1cb0 2024-03-23T14:37:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.717+0000 7f5a8a7fc640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a7806b810 0x7f5a7806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.718+0000 7f5a8a7fc640 1 -- 172.21.15.161:0/782521151 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5a90089ff0 con 0x7f5a940d1cb0 2024-03-23T14:37:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.718+0000 7f5a99384640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a7806b810 0x7f5a7806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:06.715 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.718+0000 7f5a99384640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a7806b810 0x7f5a7806dcd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5a940ba150 tx=0x7f5a84008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:06.717 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.722+0000 7f5a8a7fc640 1 -- 172.21.15.161:0/782521151 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5a90054980 con 0x7f5a940d1cb0 2024-03-23T14:37:06.922 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.927+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5a9404f550 con 0x7f5a940d1cb0 2024-03-23T14:37:06.923 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.927+0000 7f5a8a7fc640 1 -- 172.21.15.161:0/782521151 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f5a90015020 con 0x7f5a940d1cb0 2024-03-23T14:37:06.923 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a7806b810 msgr2=0x7f5a7806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a7806b810 0x7f5a7806dcd0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f5a940ba150 tx=0x7f5a84008040 comp rx=0 tx=0).stop 2024-03-23T14:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 msgr2=0x7f5a9415dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a9415dcc0 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7f5a9000b3f0 tx=0x7f5a9000b8b0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 shutdown_connections 2024-03-23T14:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a7806b810 0x7f5a7806dcd0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a94161750 0x7f5a940cc2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.926 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a940d2cd0 0x7f5a9415a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 --2- 172.21.15.161:0/782521151 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a940d1cb0 0x7f5a9415dcc0 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 >> 172.21.15.161:0/782521151 conn(0x7f5a9409d450 msgr2=0x7f5a9406d2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 shutdown_connections 2024-03-23T14:37:06.927 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:06.930+0000 7f5a9bde9640 1 -- 172.21.15.161:0/782521151 wait complete. 2024-03-23T14:37:06.940 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:37:07.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.047+0000 7fc1617c7640 1 Processor -- start 2024-03-23T14:37:07.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.047+0000 7fc1617c7640 1 -- start start 2024-03-23T14:37:07.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc1617c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc15c161820 0x7fc15c0d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc1617c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc15c161c00 con 0x7fc15c0d2ba0 2024-03-23T14:37:07.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc1617c7640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc15c0cea30 con 0x7fc15c0d1270 2024-03-23T14:37:07.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc1617c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc15c0cfd70 con 0x7fc15c161820 2024-03-23T14:37:07.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc15b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc15c0d2ba0 0x7fc15c0b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42926/0 (socket says 172.21.15.161:42926) 2024-03-23T14:37:07.045 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc15b7fe640 1 -- 172.21.15.161:0/2720128348 learned_addr learned my addr 172.21.15.161:0/2720128348 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:07.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc15affd640 1 --2- 172.21.15.161:0/2720128348 >> v2:172.21.15.183:3300/0 conn(0x7fc15c161820 0x7fc15c0d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc15a7fc640 1 -- 172.21.15.161:0/2720128348 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 231349350 0 0) 0x7fc15c161c00 con 0x7fc15c0d2ba0 2024-03-23T14:37:07.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc15a7fc640 1 -- 172.21.15.161:0/2720128348 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc148003520 con 0x7fc15c0d2ba0 2024-03-23T14:37:07.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc15affd640 1 -- 172.21.15.161:0/2720128348 >> v1:172.21.15.183:6789/0 conn(0x7fc15c0d1270 legacy=0x7fc15c0d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc15affd640 1 -- 172.21.15.161:0/2720128348 >> v1:172.21.15.161:6789/0 conn(0x7fc15c0d2ba0 legacy=0x7fc15c0b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.046 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.049+0000 7fc15affd640 1 -- 172.21.15.161:0/2720128348 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc15c0cecd0 con 0x7fc15c161820 2024-03-23T14:37:07.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.050+0000 7fc15affd640 1 --2- 172.21.15.161:0/2720128348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 0x7fc15c0d47e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0097a0 tx=0x7fc14c02b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=583b95777eb6bc25 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.050+0000 7fc15a7fc640 1 -- 172.21.15.161:0/2720128348 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c030610 con 0x7fc15c161820 2024-03-23T14:37:07.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.050+0000 7fc15a7fc640 1 -- 172.21.15.161:0/2720128348 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14c030770 con 0x7fc15c161820 2024-03-23T14:37:07.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.050+0000 7fc15a7fc640 1 -- 172.21.15.161:0/2720128348 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c030b00 con 0x7fc15c161820 2024-03-23T14:37:07.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.051+0000 7fc1617c7640 1 -- 172.21.15.161:0/2720128348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 msgr2=0x7fc15c0d47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.051+0000 7fc1617c7640 1 --2- 172.21.15.161:0/2720128348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 0x7fc15c0d47e0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0097a0 tx=0x7fc14c02b960 comp rx=0 tx=0).stop 2024-03-23T14:37:07.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.051+0000 7fc1617c7640 1 -- 172.21.15.161:0/2720128348 shutdown_connections 2024-03-23T14:37:07.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.051+0000 7fc1617c7640 1 --2- 172.21.15.161:0/2720128348 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 0x7fc15c0d47e0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.051+0000 7fc1617c7640 1 -- 172.21.15.161:0/2720128348 >> 172.21.15.161:0/2720128348 conn(0x7fc15c09d190 msgr2=0x7fc15c0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:07.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.051+0000 7fc1617c7640 1 -- 172.21.15.161:0/2720128348 shutdown_connections 2024-03-23T14:37:07.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.051+0000 7fc1617c7640 1 -- 172.21.15.161:0/2720128348 wait complete. 2024-03-23T14:37:07.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.052+0000 7fc1617c7640 1 Processor -- start 2024-03-23T14:37:07.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.052+0000 7fc1617c7640 1 -- start start 2024-03-23T14:37:07.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc1617c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0d1270 0x7fc15c0b4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc1617c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc15c0d2ba0 0x7fc15c15c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc1617c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 0x7fc15c0b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc1617c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c0b9210 con 0x7fc15c0d1270 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc1617c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc15c0b4c90 con 0x7fc15c161820 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc1617c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc15c0b36b0 con 0x7fc15c0d2ba0 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0d1270 0x7fc15c0b4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0d1270 0x7fc15c0b4750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45280/0 (socket says 172.21.15.161:45280) 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 -- 172.21.15.161:0/2724623175 learned_addr learned my addr 172.21.15.161:0/2724623175 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15b7fe640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 0x7fc15c0b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc153fff640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc15c0d2ba0 0x7fc15c15c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 -- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc15c0d2ba0 msgr2=0x7fc15c15c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc15c0d2ba0 0x7fc15c15c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 -- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 msgr2=0x7fc15c0b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 0x7fc15c0b8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 -- 172.21.15.161:0/2724623175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc14c009400 con 0x7fc15c0d1270 2024-03-23T14:37:07.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.053+0000 7fc15affd640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0d1270 0x7fc15c0b4750 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0098c0 tx=0x7fc14c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.054+0000 7fc1537fe640 1 -- 172.21.15.161:0/2724623175 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c03e040 con 0x7fc15c0d1270 2024-03-23T14:37:07.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.054+0000 7fc1537fe640 1 -- 172.21.15.161:0/2724623175 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc14c039cb0 con 0x7fc15c0d1270 2024-03-23T14:37:07.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.054+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc15c0b3990 con 0x7fc15c0d1270 2024-03-23T14:37:07.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.054+0000 7fc1537fe640 1 -- 172.21.15.161:0/2724623175 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc14c044630 con 0x7fc15c0d1270 2024-03-23T14:37:07.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.054+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc15c0b3f50 con 0x7fc15c0d1270 2024-03-23T14:37:07.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.055+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc15c04f4e0 con 0x7fc15c0d1270 2024-03-23T14:37:07.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.055+0000 7fc1537fe640 1 -- 172.21.15.161:0/2724623175 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc14c04d3b0 con 0x7fc15c0d1270 2024-03-23T14:37:07.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.055+0000 7fc1537fe640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc13c06b810 0x7fc13c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.056+0000 7fc1537fe640 1 -- 172.21.15.161:0/2724623175 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fc14c0ac650 con 0x7fc15c0d1270 2024-03-23T14:37:07.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.056+0000 7fc153fff640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc13c06b810 0x7fc13c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.053 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.056+0000 7fc153fff640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc13c06b810 0x7fc13c06dcd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc15c1f7760 tx=0x7fc144009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.055 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.060+0000 7fc1537fe640 1 -- 172.21.15.161:0/2724623175 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc14c077090 con 0x7fc15c0d1270 2024-03-23T14:37:07.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.245+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc15c0d45c0 con 0x7fc15c0d1270 2024-03-23T14:37:07.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.246+0000 7fc1537fe640 1 -- 172.21.15.161:0/2724623175 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7fc14c07a750 con 0x7fc15c0d1270 2024-03-23T14:37:07.242 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc13c06b810 msgr2=0x7fc13c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc13c06b810 0x7fc13c06dcd0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc15c1f7760 tx=0x7fc144009040 comp rx=0 tx=0).stop 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0d1270 msgr2=0x7fc15c0b4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0d1270 0x7fc15c0b4750 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7fc14c0098c0 tx=0x7fc14c004730 comp rx=0 tx=0).stop 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 shutdown_connections 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc13c06b810 0x7fc13c06dcd0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc15c161820 0x7fc15c0b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc15c0d2ba0 0x7fc15c15c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 --2- 172.21.15.161:0/2724623175 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc15c0d1270 0x7fc15c0b4750 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 >> 172.21.15.161:0/2724623175 conn(0x7fc15c09d190 msgr2=0x7fc15c0b21f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:07.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 shutdown_connections 2024-03-23T14:37:07.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.248+0000 7fc1617c7640 1 -- 172.21.15.161:0/2724623175 wait complete. 2024-03-23T14:37:07.254 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:37:07.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.361+0000 7f5dc953e640 1 Processor -- start 2024-03-23T14:37:07.357 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.361+0000 7f5dc953e640 1 -- start start 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc953e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5dc415e400 0x7f5dc40cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc953e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5dc40d2b40 0x7f5dc40b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc953e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5dc40d0460 con 0x7f5dc40cf9d0 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc953e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5dc40d1050 con 0x7f5dc40d2b40 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc953e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f5dc40d06b0 con 0x7f5dc415e400 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc37fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5dc40d2b40 0x7f5dc40b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc27fc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5dc40cf9d0 0x7f5dc40d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42934/0 (socket says 172.21.15.161:42934) 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc27fc640 1 -- 172.21.15.161:0/3347619526 learned_addr learned my addr 172.21.15.161:0/3347619526 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc2ffd640 1 --2- 172.21.15.161:0/3347619526 >> v2:172.21.15.183:3300/0 conn(0x7f5dc415e400 0x7f5dc40cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc37fe640 1 -- 172.21.15.161:0/3347619526 >> v2:172.21.15.183:3300/0 conn(0x7f5dc415e400 msgr2=0x7f5dc40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.362+0000 7f5dc37fe640 1 --2- 172.21.15.161:0/3347619526 >> v2:172.21.15.183:3300/0 conn(0x7f5dc415e400 0x7f5dc40cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.363+0000 7f5dc37fe640 1 -- 172.21.15.161:0/3347619526 >> v1:172.21.15.161:6789/0 conn(0x7f5dc40cf9d0 legacy=0x7f5dc40d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.363+0000 7f5dc37fe640 1 -- 172.21.15.161:0/3347619526 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dc40d0990 con 0x7f5dc40d2b40 2024-03-23T14:37:07.359 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.363+0000 7f5dc2ffd640 1 --2- 172.21.15.161:0/3347619526 >> v2:172.21.15.183:3300/0 conn(0x7f5dc415e400 0x7f5dc40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:07.360 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.363+0000 7f5dc37fe640 1 --2- 172.21.15.161:0/3347619526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc40d2b40 0x7f5dc40b0a80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5dc408e230 tx=0x7f5dbc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a256cf115910419f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.363+0000 7f5dc1ffb640 1 -- 172.21.15.161:0/3347619526 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc02cc80 con 0x7f5dc40d2b40 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.363+0000 7f5dc1ffb640 1 -- 172.21.15.161:0/3347619526 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc02cde0 con 0x7f5dc40d2b40 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.363+0000 7f5dc1ffb640 1 -- 172.21.15.161:0/3347619526 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc034660 con 0x7f5dc40d2b40 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.364+0000 7f5dc953e640 1 -- 172.21.15.161:0/3347619526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc40d2b40 msgr2=0x7f5dc40b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.364+0000 7f5dc953e640 1 --2- 172.21.15.161:0/3347619526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc40d2b40 0x7f5dc40b0a80 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7f5dc408e230 tx=0x7f5dbc02bed0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.364+0000 7f5dc953e640 1 -- 172.21.15.161:0/3347619526 shutdown_connections 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.364+0000 7f5dc953e640 1 --2- 172.21.15.161:0/3347619526 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc40d2b40 0x7f5dc40b0a80 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.364+0000 7f5dc953e640 1 --2- 172.21.15.161:0/3347619526 >> v2:172.21.15.183:3300/0 conn(0x7f5dc415e400 0x7f5dc40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.364+0000 7f5dc953e640 1 -- 172.21.15.161:0/3347619526 >> 172.21.15.161:0/3347619526 conn(0x7f5dc409d190 msgr2=0x7f5dc40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.364+0000 7f5dc953e640 1 -- 172.21.15.161:0/3347619526 shutdown_connections 2024-03-23T14:37:07.361 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.364+0000 7f5dc953e640 1 -- 172.21.15.161:0/3347619526 wait complete. 2024-03-23T14:37:07.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.366+0000 7f5dc953e640 1 Processor -- start 2024-03-23T14:37:07.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.366+0000 7f5dc953e640 1 -- start start 2024-03-23T14:37:07.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc953e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc40cf9d0 0x7f5dc41523d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc953e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc40d2b40 0x7f5dc40b9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc953e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc415e400 0x7f5dc415ac10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc953e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc4152910 con 0x7f5dc415e400 2024-03-23T14:37:07.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc953e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5dc40ccf80 con 0x7f5dc40d2b40 2024-03-23T14:37:07.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc953e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5dc40cd190 con 0x7f5dc40cf9d0 2024-03-23T14:37:07.364 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc415e400 0x7f5dc415ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc415e400 0x7f5dc415ac10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45300/0 (socket says 172.21.15.161:45300) 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.367+0000 7f5dc37fe640 1 -- 172.21.15.161:0/2090706745 learned_addr learned my addr 172.21.15.161:0/2090706745 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc37fe640 1 -- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc40cf9d0 msgr2=0x7f5dc41523d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc27fc640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc40d2b40 0x7f5dc40b9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc2ffd640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc40cf9d0 0x7f5dc41523d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc37fe640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc40cf9d0 0x7f5dc41523d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc37fe640 1 -- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc40d2b40 msgr2=0x7f5dc40b9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc37fe640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc40d2b40 0x7f5dc40b9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc37fe640 1 -- 172.21.15.161:0/2090706745 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dbc009a70 con 0x7f5dc415e400 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc27fc640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc40d2b40 0x7f5dc40b9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc37fe640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc415e400 0x7f5dc415ac10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009e80 tx=0x7f5dbc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dab7fe640 1 -- 172.21.15.161:0/2090706745 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc03e610 con 0x7f5dc415e400 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dab7fe640 1 -- 172.21.15.161:0/2090706745 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dbc03e770 con 0x7f5dc415e400 2024-03-23T14:37:07.365 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dab7fe640 1 -- 172.21.15.161:0/2090706745 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dbc03eb20 con 0x7f5dc415e400 2024-03-23T14:37:07.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dc40cd470 con 0x7f5dc415e400 2024-03-23T14:37:07.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.368+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dc40cda30 con 0x7f5dc415e400 2024-03-23T14:37:07.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.369+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dc40a1e00 con 0x7f5dc415e400 2024-03-23T14:37:07.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.370+0000 7f5dab7fe640 1 -- 172.21.15.161:0/2090706745 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5dbc03b050 con 0x7f5dc415e400 2024-03-23T14:37:07.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.370+0000 7f5dab7fe640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5db006b810 0x7f5db006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.370+0000 7f5dab7fe640 1 -- 172.21.15.161:0/2090706745 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5dbc0acb90 con 0x7f5dc415e400 2024-03-23T14:37:07.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.370+0000 7f5dc2ffd640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5db006b810 0x7f5db006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.371+0000 7f5dc2ffd640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5db006b810 0x7f5db006dcd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5db400b840 tx=0x7f5db4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.370 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.374+0000 7f5dab7fe640 1 -- 172.21.15.161:0/2090706745 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5dbc077670 con 0x7f5dc415e400 2024-03-23T14:37:07.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.564+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f5dc404f550 con 0x7f5dc415e400 2024-03-23T14:37:07.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.565+0000 7f5dab7fe640 1 -- 172.21.15.161:0/2090706745 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) ==== 80+0+0 (secure 0 0 0) 0x7f5dbc04e080 con 0x7f5dc415e400 2024-03-23T14:37:07.561 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5db006b810 msgr2=0x7f5db006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5db006b810 0x7f5db006dcd0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5db400b840 tx=0x7f5db4008040 comp rx=0 tx=0).stop 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc415e400 msgr2=0x7f5dc415ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc415e400 0x7f5dc415ac10 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7f5dbc009e80 tx=0x7f5dbc004a40 comp rx=0 tx=0).stop 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 shutdown_connections 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5db006b810 0x7f5db006dcd0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5dc415e400 0x7f5dc415ac10 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5dc40d2b40 0x7f5dc40b9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 --2- 172.21.15.161:0/2090706745 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5dc40cf9d0 0x7f5dc41523d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 >> 172.21.15.161:0/2090706745 conn(0x7f5dc409d190 msgr2=0x7f5dc40b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 shutdown_connections 2024-03-23T14:37:07.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.567+0000 7f5dc953e640 1 -- 172.21.15.161:0/2090706745 wait complete. 2024-03-23T14:37:07.574 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:37:07.574 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:07.582 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:37:07.589 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:07.590 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:37:07.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.735+0000 7fceba664640 1 Processor -- start 2024-03-23T14:37:07.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.735+0000 7fceba664640 1 -- start start 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.736+0000 7fceba664640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fceb40d1470 0x7fceb40d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.736+0000 7fceba664640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb40d0460 con 0x7fceb415e400 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.736+0000 7fceba664640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fceb40d06b0 con 0x7fceb40d2970 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.736+0000 7fceba664640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fceb40d1050 con 0x7fceb40d1470 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.736+0000 7fceb3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fceb415e400 0x7fceb40cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42948/0 (socket says 172.21.15.161:42948) 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.736+0000 7fceb3fff640 1 -- 172.21.15.161:0/2917874610 learned_addr learned my addr 172.21.15.161:0/2917874610 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.737+0000 7fceb37fe640 1 --2- 172.21.15.161:0/2917874610 >> v2:172.21.15.183:3300/0 conn(0x7fceb40d1470 0x7fceb40d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.737+0000 7fceb2ffd640 1 -- 172.21.15.161:0/2917874610 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 529589866 0 0) 0x7fceb40d0460 con 0x7fceb415e400 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.737+0000 7fceb2ffd640 1 -- 172.21.15.161:0/2917874610 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce9c003520 con 0x7fceb415e400 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.737+0000 7fceb2ffd640 1 -- 172.21.15.161:0/2917874610 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1225272603 0 0) 0x7fceb40d06b0 con 0x7fceb40d2970 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.737+0000 7fceb2ffd640 1 -- 172.21.15.161:0/2917874610 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceb40d0460 con 0x7fceb40d2970 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.737+0000 7fceb37fe640 1 -- 172.21.15.161:0/2917874610 >> v1:172.21.15.183:6789/0 conn(0x7fceb40d2970 legacy=0x7fceb40b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.733 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.737+0000 7fceb37fe640 1 -- 172.21.15.161:0/2917874610 >> v1:172.21.15.161:6789/0 conn(0x7fceb415e400 legacy=0x7fceb40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.734 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.737+0000 7fceb37fe640 1 -- 172.21.15.161:0/2917874610 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fceb40d0990 con 0x7fceb40d1470 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.738+0000 7fceb37fe640 1 --2- 172.21.15.161:0/2917874610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 0x7fceb40d2430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcea80097a0 tx=0x7fcea802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fa466a5260d72fc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.738+0000 7fceb2ffd640 1 -- 172.21.15.161:0/2917874610 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8030610 con 0x7fceb40d1470 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.738+0000 7fceb2ffd640 1 -- 172.21.15.161:0/2917874610 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea8030770 con 0x7fceb40d1470 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.738+0000 7fceb2ffd640 1 -- 172.21.15.161:0/2917874610 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8030b00 con 0x7fceb40d1470 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.738+0000 7fceba664640 1 -- 172.21.15.161:0/2917874610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 msgr2=0x7fceb40d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.738+0000 7fceba664640 1 --2- 172.21.15.161:0/2917874610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 0x7fceb40d2430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcea80097a0 tx=0x7fcea802b960 comp rx=0 tx=0).stop 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.739+0000 7fceba664640 1 -- 172.21.15.161:0/2917874610 shutdown_connections 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.739+0000 7fceba664640 1 --2- 172.21.15.161:0/2917874610 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 0x7fceb40d2430 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.739+0000 7fceba664640 1 -- 172.21.15.161:0/2917874610 >> 172.21.15.161:0/2917874610 conn(0x7fceb409d190 msgr2=0x7fceb40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.739+0000 7fceba664640 1 -- 172.21.15.161:0/2917874610 shutdown_connections 2024-03-23T14:37:07.735 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.739+0000 7fceba664640 1 -- 172.21.15.161:0/2917874610 wait complete. 2024-03-23T14:37:07.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.740+0000 7fceba664640 1 Processor -- start 2024-03-23T14:37:07.736 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.741+0000 7fceba664640 1 -- start start 2024-03-23T14:37:07.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceba664640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 0x7fceb40c8300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceba664640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb40d2970 0x7fceb41f4680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceba664640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb415e400 0x7fceb40b34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceba664640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fceb41f4bc0 con 0x7fceb40d2970 2024-03-23T14:37:07.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceba664640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fceb40c89e0 con 0x7fceb40d1470 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceba664640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fceb40c8cc0 con 0x7fceb415e400 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb40d2970 0x7fceb41f4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb40d2970 0x7fceb41f4680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45304/0 (socket says 172.21.15.161:45304) 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 -- 172.21.15.161:0/3322030411 learned_addr learned my addr 172.21.15.161:0/3322030411 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb3fff640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 0x7fceb40c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 -- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb415e400 msgr2=0x7fceb40b34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb8c01640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb415e400 0x7fceb40b34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb415e400 0x7fceb40b34d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 -- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 msgr2=0x7fceb40c8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 0x7fceb40c8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 -- 172.21.15.161:0/3322030411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcea8009400 con 0x7fceb40d2970 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb3fff640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 0x7fceb40c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.742+0000 7fceb37fe640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb40d2970 0x7fceb41f4680 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcea80098c0 tx=0x7fcea80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.743+0000 7fceb0ff9640 1 -- 172.21.15.161:0/3322030411 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea803e070 con 0x7fceb40d2970 2024-03-23T14:37:07.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.743+0000 7fceb0ff9640 1 -- 172.21.15.161:0/3322030411 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcea80397a0 con 0x7fceb40d2970 2024-03-23T14:37:07.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.743+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fceb40c8fa0 con 0x7fceb40d2970 2024-03-23T14:37:07.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.743+0000 7fceb0ff9640 1 -- 172.21.15.161:0/3322030411 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcea8039b70 con 0x7fceb40d2970 2024-03-23T14:37:07.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.743+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fceb415ea10 con 0x7fceb40d2970 2024-03-23T14:37:07.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.744+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fceb415ec00 con 0x7fceb40d2970 2024-03-23T14:37:07.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.744+0000 7fceb0ff9640 1 -- 172.21.15.161:0/3322030411 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcea8039cd0 con 0x7fceb40d2970 2024-03-23T14:37:07.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.744+0000 7fceb0ff9640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcea006b810 0x7fcea006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:07.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.745+0000 7fceb0ff9640 1 -- 172.21.15.161:0/3322030411 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fcea80ac4b0 con 0x7fceb40d2970 2024-03-23T14:37:07.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.745+0000 7fceb3fff640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcea006b810 0x7fcea006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:07.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.745+0000 7fceb3fff640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcea006b810 0x7fcea006dcd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcea4000f60 tx=0x7fcea4009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:07.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.749+0000 7fceb0ff9640 1 -- 172.21.15.161:0/3322030411 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcea8076ed0 con 0x7fceb40d2970 2024-03-23T14:37:07.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.958+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fceb40a1e00 con 0x7fceb40d2970 2024-03-23T14:37:07.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.959+0000 7fceb0ff9640 1 -- 172.21.15.161:0/3322030411 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7fcea807a590 con 0x7fceb40d2970 2024-03-23T14:37:07.955 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:07.955 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:07.957 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcea006b810 msgr2=0x7fcea006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcea006b810 0x7fcea006dcd0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fcea4000f60 tx=0x7fcea4009040 comp rx=0 tx=0).stop 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb40d2970 msgr2=0x7fceb41f4680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb40d2970 0x7fceb41f4680 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fcea80098c0 tx=0x7fcea80046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 shutdown_connections 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcea006b810 0x7fcea006dcd0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fceb415e400 0x7fceb40b34d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fceb40d2970 0x7fceb41f4680 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 --2- 172.21.15.161:0/3322030411 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fceb40d1470 0x7fceb40c8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.961+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 >> 172.21.15.161:0/3322030411 conn(0x7fceb409d190 msgr2=0x7fceb406cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.962+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 shutdown_connections 2024-03-23T14:37:07.958 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:07.962+0000 7fceba664640 1 -- 172.21.15.161:0/3322030411 wait complete. 2024-03-23T14:37:07.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:56.747667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:57.478793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:37:07.969 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:37:08.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.077+0000 7f90ba223640 1 Processor -- start 2024-03-23T14:37:08.073 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90ba223640 1 -- start start 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90ba223640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f90b4161820 0x7f90b40d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90ba223640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f90b40d1270 0x7f90b40d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90ba223640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d29d0 0x7f90b4161e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90ba223640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f90b4161c00 con 0x7f90b40d1270 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90ba223640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f90b40d2e20 con 0x7f90b4161820 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90ba223640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f90b40cea30 con 0x7f90b40d29d0 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90b2ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f90b40d1270 0x7f90b40d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90b2ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f90b40d1270 0x7f90b40d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45306/0 (socket says 172.21.15.161:45306) 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.078+0000 7f90b2ffd640 1 -- 172.21.15.161:0/2779038789 learned_addr learned my addr 172.21.15.161:0/2779038789 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.079+0000 7f90b2ffd640 1 -- 172.21.15.161:0/2779038789 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d29d0 msgr2=0x7f90b4161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.079+0000 7f90b2ffd640 1 --2- 172.21.15.161:0/2779038789 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d29d0 0x7f90b4161e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.075 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.079+0000 7f90b2ffd640 1 -- 172.21.15.161:0/2779038789 >> v2:172.21.15.183:3300/0 conn(0x7f90b4161820 msgr2=0x7f90b40d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:08.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.079+0000 7f90b2ffd640 1 --2- 172.21.15.161:0/2779038789 >> v2:172.21.15.183:3300/0 conn(0x7f90b4161820 0x7f90b40d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.076 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.079+0000 7f90b2ffd640 1 -- 172.21.15.161:0/2779038789 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90b40cfd70 con 0x7f90b40d1270 2024-03-23T14:37:08.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.080+0000 7f90b2ffd640 1 --2- 172.21.15.161:0/2779038789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d1270 0x7f90b40d2490 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f90b406b270 tx=0x7f90a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5796b199ee7284f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.080+0000 7f90b27fc640 1 -- 172.21.15.161:0/2779038789 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8031870 con 0x7f90b40d1270 2024-03-23T14:37:08.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.080+0000 7f90b27fc640 1 -- 172.21.15.161:0/2779038789 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a80319d0 con 0x7f90b40d1270 2024-03-23T14:37:08.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.080+0000 7f90b27fc640 1 -- 172.21.15.161:0/2779038789 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8031d60 con 0x7f90b40d1270 2024-03-23T14:37:08.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.080+0000 7f90ba223640 1 -- 172.21.15.161:0/2779038789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d1270 msgr2=0x7f90b40d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.077 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.081+0000 7f90ba223640 1 --2- 172.21.15.161:0/2779038789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d1270 0x7f90b40d2490 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f90b406b270 tx=0x7f90a802b910 comp rx=0 tx=0).stop 2024-03-23T14:37:08.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.081+0000 7f90ba223640 1 -- 172.21.15.161:0/2779038789 shutdown_connections 2024-03-23T14:37:08.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.081+0000 7f90ba223640 1 --2- 172.21.15.161:0/2779038789 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d29d0 0x7f90b4161e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.081+0000 7f90ba223640 1 --2- 172.21.15.161:0/2779038789 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d1270 0x7f90b40d2490 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.081+0000 7f90ba223640 1 --2- 172.21.15.161:0/2779038789 >> v2:172.21.15.183:3300/0 conn(0x7f90b4161820 0x7f90b40d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.081+0000 7f90ba223640 1 -- 172.21.15.161:0/2779038789 >> 172.21.15.161:0/2779038789 conn(0x7f90b409d190 msgr2=0x7f90b40d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:08.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.081+0000 7f90ba223640 1 -- 172.21.15.161:0/2779038789 shutdown_connections 2024-03-23T14:37:08.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.081+0000 7f90ba223640 1 -- 172.21.15.161:0/2779038789 wait complete. 2024-03-23T14:37:08.078 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.082+0000 7f90ba223640 1 Processor -- start 2024-03-23T14:37:08.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.082+0000 7f90ba223640 1 -- start start 2024-03-23T14:37:08.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90ba223640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d1270 0x7f90b40b69f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90ba223640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d29d0 0x7f90b415c660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90ba223640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b4161820 0x7f90b40b4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90ba223640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f90b40cc200 con 0x7f90b40d29d0 2024-03-23T14:37:08.081 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90ba223640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f90b40b7d50 con 0x7f90b4161820 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90ba223640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f90b40b6f30 con 0x7f90b40d1270 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d29d0 0x7f90b415c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d29d0 0x7f90b415c660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45312/0 (socket says 172.21.15.161:45312) 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 -- 172.21.15.161:0/2144878204 learned_addr learned my addr 172.21.15.161:0/2144878204 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b37fe640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d1270 0x7f90b40b69f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b3fff640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b4161820 0x7f90b40b4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 -- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d1270 msgr2=0x7f90b40b69f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d1270 0x7f90b40b69f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 -- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b4161820 msgr2=0x7f90b40b4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b4161820 0x7f90b40b4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 -- 172.21.15.161:0/2144878204 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90a80094d0 con 0x7f90b40d29d0 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b3fff640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b4161820 0x7f90b40b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:08.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90b2ffd640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d29d0 0x7f90b415c660 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f90a8000c00 tx=0x7f90a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f909bfff640 1 -- 172.21.15.161:0/2144878204 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a803ad10 con 0x7f90b40d29d0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f909bfff640 1 -- 172.21.15.161:0/2144878204 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f90a803ae70 con 0x7f90b40d29d0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90b40b71d0 con 0x7f90b40d29d0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f909bfff640 1 -- 172.21.15.161:0/2144878204 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f90a8044610 con 0x7f90b40d29d0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.083+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90b40b7790 con 0x7f90b40d29d0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.084+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f90b40cc370 con 0x7f90b40d29d0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.085+0000 7f909bfff640 1 -- 172.21.15.161:0/2144878204 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f90a804d450 con 0x7f90b40d29d0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.085+0000 7f909bfff640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f90a006b810 0x7f90a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.085+0000 7f909bfff640 1 -- 172.21.15.161:0/2144878204 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f90a802d040 con 0x7f90b40d29d0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.085+0000 7f90b37fe640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f90a006b810 0x7f90a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.086+0000 7f90b37fe640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f90a006b810 0x7f90a006dcd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f90a400a810 tx=0x7f90a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.090+0000 7f909bfff640 1 -- 172.21.15.161:0/2144878204 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f90a80772e0 con 0x7f90b40d29d0 2024-03-23T14:37:08.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.264+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f90b404f4e0 con 0x7f90b40d29d0 2024-03-23T14:37:08.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.265+0000 7f909bfff640 1 -- 172.21.15.161:0/2144878204 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f90a807a9a0 con 0x7f90b40d29d0 2024-03-23T14:37:08.261 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f90a006b810 msgr2=0x7f90a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f90a006b810 0x7f90a006dcd0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f90a400a810 tx=0x7f90a4008040 comp rx=0 tx=0).stop 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d29d0 msgr2=0x7f90b415c660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d29d0 0x7f90b415c660 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f90a8000c00 tx=0x7f90a80046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 shutdown_connections 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f90a006b810 0x7f90a006dcd0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f90b4161820 0x7f90b40b4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f90b40d29d0 0x7f90b415c660 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 --2- 172.21.15.161:0/2144878204 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f90b40d1270 0x7f90b40b69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 >> 172.21.15.161:0/2144878204 conn(0x7f90b409d190 msgr2=0x7f90b40b1400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:08.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 shutdown_connections 2024-03-23T14:37:08.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.267+0000 7f90ba223640 1 -- 172.21.15.161:0/2144878204 wait complete. 2024-03-23T14:37:08.274 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:37:08.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.381+0000 7f4042464640 1 Processor -- start 2024-03-23T14:37:08.377 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.381+0000 7f4042464640 1 -- start start 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.382+0000 7f4042464640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f403c0d2cd0 0x7f403c0ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.382+0000 7f4042464640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.382+0000 7f4042464640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f403c0d1330 con 0x7f403c161750 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.382+0000 7f4042464640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f403c0d0720 con 0x7f403c0d2cd0 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.382+0000 7f4042464640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f403c0d0a00 con 0x7f403c0d1cb0 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.382+0000 7f403b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f403c0d2cd0 0x7f403c0ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.382+0000 7f403b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f403c0d2cd0 0x7f403c0ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45314/0 (socket says 172.21.15.161:45314) 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.382+0000 7f403b7fe640 1 -- 172.21.15.161:0/3191626514 learned_addr learned my addr 172.21.15.161:0/3191626514 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:08.379 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.383+0000 7f4040a01640 1 --2- 172.21.15.161:0/3191626514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.383+0000 7f403b7fe640 1 -- 172.21.15.161:0/3191626514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 msgr2=0x7f403c0cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.383+0000 7f403b7fe640 1 --2- 172.21.15.161:0/3191626514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.383+0000 7f403b7fe640 1 -- 172.21.15.161:0/3191626514 >> v1:172.21.15.183:6789/0 conn(0x7f403c161750 legacy=0x7f403c0d25c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.383+0000 7f403b7fe640 1 -- 172.21.15.161:0/3191626514 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f403c0d0ce0 con 0x7f403c0d2cd0 2024-03-23T14:37:08.380 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.383+0000 7f4040a01640 1 --2- 172.21.15.161:0/3191626514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:08.381 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f403b7fe640 1 --2- 172.21.15.161:0/3191626514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0ceb10 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f4030009870 tx=0x7f403002b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b6374866f6cbc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f403affd640 1 -- 172.21.15.161:0/3191626514 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030030840 con 0x7f403c0d2cd0 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f403affd640 1 -- 172.21.15.161:0/3191626514 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f40300309a0 con 0x7f403c0d2cd0 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f403affd640 1 -- 172.21.15.161:0/3191626514 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030030d30 con 0x7f403c0d2cd0 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f4042464640 1 -- 172.21.15.161:0/3191626514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 msgr2=0x7f403c0ceb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f4042464640 1 --2- 172.21.15.161:0/3191626514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0ceb10 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f4030009870 tx=0x7f403002b980 comp rx=0 tx=0).stop 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f4042464640 1 -- 172.21.15.161:0/3191626514 shutdown_connections 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f4042464640 1 --2- 172.21.15.161:0/3191626514 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f4042464640 1 --2- 172.21.15.161:0/3191626514 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0ceb10 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f4042464640 1 -- 172.21.15.161:0/3191626514 >> 172.21.15.161:0/3191626514 conn(0x7f403c09d450 msgr2=0x7f403c0d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f4042464640 1 -- 172.21.15.161:0/3191626514 shutdown_connections 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.384+0000 7f4042464640 1 -- 172.21.15.161:0/3191626514 wait complete. 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.385+0000 7f4042464640 1 Processor -- start 2024-03-23T14:37:08.382 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.386+0000 7f4042464640 1 -- start start 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.386+0000 7f4042464640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cde30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.386+0000 7f4042464640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0b6cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.386+0000 7f4042464640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f403c161750 0x7f403c15c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.386+0000 7f4042464640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f403c15d790 con 0x7f403c0d2cd0 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.386+0000 7f4042464640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f403c0b7360 con 0x7f403c161750 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.386+0000 7f4042464640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f403c0b7640 con 0x7f403c0d1cb0 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0b6cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0b6cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45328/0 (socket says 172.21.15.161:45328) 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 -- 172.21.15.161:0/3991541215 learned_addr learned my addr 172.21.15.161:0/3991541215 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:08.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403bfff640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f4040a01640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f403c161750 0x7f403c15c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 -- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 msgr2=0x7f403c0cde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 -- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f403c161750 msgr2=0x7f403c15c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f403c161750 0x7f403c15c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 -- 172.21.15.161:0/3991541215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40300094d0 con 0x7f403c0d2cd0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f403b7fe640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0b6cd0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4030000c00 tx=0x7f4030004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f4038ff9640 1 -- 172.21.15.161:0/3991541215 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4030030810 con 0x7f403c0d2cd0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f4038ff9640 1 -- 172.21.15.161:0/3991541215 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4030043400 con 0x7f403c0d2cd0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f4038ff9640 1 -- 172.21.15.161:0/3991541215 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40300437d0 con 0x7f403c0d2cd0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f403c0b7920 con 0x7f403c0d2cd0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.387+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f403c0b7ee0 con 0x7f403c0d2cd0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.389+0000 7f4038ff9640 1 -- 172.21.15.161:0/3991541215 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f4030043930 con 0x7f403c0d2cd0 2024-03-23T14:37:08.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.389+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f403c0a2030 con 0x7f403c0d2cd0 2024-03-23T14:37:08.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.389+0000 7f4038ff9640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f402806b810 0x7f402806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.389+0000 7f4038ff9640 1 -- 172.21.15.161:0/3991541215 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f403003e070 con 0x7f403c0d2cd0 2024-03-23T14:37:08.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.389+0000 7f403bfff640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f402806b810 0x7f402806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.390+0000 7f403bfff640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f402806b810 0x7f402806dcd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f403c1ea690 tx=0x7f402c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.393+0000 7f4038ff9640 1 -- 172.21.15.161:0/3991541215 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f4030077f10 con 0x7f403c0d2cd0 2024-03-23T14:37:08.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.568+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f403c0d0100 con 0x7f403c0d2cd0 2024-03-23T14:37:08.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.568+0000 7f4038ff9640 1 -- 172.21.15.161:0/3991541215 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f403007b5d0 con 0x7f403c0d2cd0 2024-03-23T14:37:08.564 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f402806b810 msgr2=0x7f402806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f402806b810 0x7f402806dcd0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f403c1ea690 tx=0x7f402c009040 comp rx=0 tx=0).stop 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 msgr2=0x7f403c0b6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0b6cd0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f4030000c00 tx=0x7f4030004730 comp rx=0 tx=0).stop 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 shutdown_connections 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f402806b810 0x7f402806dcd0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f403c161750 0x7f403c15c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f403c0d2cd0 0x7f403c0b6cd0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 --2- 172.21.15.161:0/3991541215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f403c0d1cb0 0x7f403c0cde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 >> 172.21.15.161:0/3991541215 conn(0x7f403c09d450 msgr2=0x7f403c0b1070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:08.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 shutdown_connections 2024-03-23T14:37:08.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.571+0000 7f4042464640 1 -- 172.21.15.161:0/3991541215 wait complete. 2024-03-23T14:37:08.578 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:37:08.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.690+0000 7f228916c640 1 Processor -- start 2024-03-23T14:37:08.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.690+0000 7f228916c640 1 -- start start 2024-03-23T14:37:08.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.690+0000 7f228916c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2284161820 0x7f22840d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228916c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d1270 0x7f22840d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228916c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22840d29d0 0x7f2284161e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228916c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2284161c00 con 0x7f22840d29d0 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228916c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f22840d2e20 con 0x7f2284161820 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228916c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f22840cea30 con 0x7f22840d1270 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228359e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22840d29d0 0x7f2284161e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228359e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22840d29d0 0x7f2284161e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45330/0 (socket says 172.21.15.161:45330) 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228359e640 1 -- 172.21.15.161:0/1375085082 learned_addr learned my addr 172.21.15.161:0/1375085082 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f2282d9d640 1 --2- 172.21.15.161:0/1375085082 >> v2:172.21.15.183:3300/0 conn(0x7f2284161820 0x7f22840d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f227bfff640 1 --2- 172.21.15.161:0/1375085082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d1270 0x7f22840d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228359e640 1 -- 172.21.15.161:0/1375085082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d1270 msgr2=0x7f22840d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228359e640 1 --2- 172.21.15.161:0/1375085082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d1270 0x7f22840d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228359e640 1 -- 172.21.15.161:0/1375085082 >> v2:172.21.15.183:3300/0 conn(0x7f2284161820 msgr2=0x7f22840d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228359e640 1 --2- 172.21.15.161:0/1375085082 >> v2:172.21.15.183:3300/0 conn(0x7f2284161820 0x7f22840d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.689 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f228359e640 1 -- 172.21.15.161:0/1375085082 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22840cfd70 con 0x7f22840d29d0 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.691+0000 7f2282d9d640 1 --2- 172.21.15.161:0/1375085082 >> v2:172.21.15.183:3300/0 conn(0x7f2284161820 0x7f22840d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.692+0000 7f228359e640 1 --2- 172.21.15.161:0/1375085082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d29d0 0x7f2284161e00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f228406b270 tx=0x7f227c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88f9eb39fb9f04d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.692+0000 7f228259c640 1 -- 172.21.15.161:0/1375085082 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c02ca70 con 0x7f22840d29d0 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.692+0000 7f228259c640 1 -- 172.21.15.161:0/1375085082 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f227c02cbd0 con 0x7f22840d29d0 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.692+0000 7f228259c640 1 -- 172.21.15.161:0/1375085082 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f227c0343f0 con 0x7f22840d29d0 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.692+0000 7f228916c640 1 -- 172.21.15.161:0/1375085082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d29d0 msgr2=0x7f2284161e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.692+0000 7f228916c640 1 --2- 172.21.15.161:0/1375085082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d29d0 0x7f2284161e00 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f228406b270 tx=0x7f227c02bed0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.693+0000 7f228916c640 1 -- 172.21.15.161:0/1375085082 shutdown_connections 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.693+0000 7f228916c640 1 --2- 172.21.15.161:0/1375085082 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d29d0 0x7f2284161e00 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.693+0000 7f228916c640 1 --2- 172.21.15.161:0/1375085082 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d1270 0x7f22840d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.693+0000 7f228916c640 1 --2- 172.21.15.161:0/1375085082 >> v2:172.21.15.183:3300/0 conn(0x7f2284161820 0x7f22840d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.693+0000 7f228916c640 1 -- 172.21.15.161:0/1375085082 >> 172.21.15.161:0/1375085082 conn(0x7f228409d190 msgr2=0x7f22840d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.693+0000 7f228916c640 1 -- 172.21.15.161:0/1375085082 shutdown_connections 2024-03-23T14:37:08.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.693+0000 7f228916c640 1 -- 172.21.15.161:0/1375085082 wait complete. 2024-03-23T14:37:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.694+0000 7f228916c640 1 Processor -- start 2024-03-23T14:37:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.694+0000 7f228916c640 1 -- start start 2024-03-23T14:37:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228916c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d1270 0x7f22841599f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228916c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d29d0 0x7f22840c79e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228916c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2284161820 0x7f228415f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228916c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22840ca0f0 con 0x7f22840d1270 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228916c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f22840b6ec0 con 0x7f2284161820 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228916c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2284159f30 con 0x7f22840d29d0 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d1270 0x7f22841599f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d1270 0x7f22841599f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45332/0 (socket says 172.21.15.161:45332) 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 -- 172.21.15.161:0/206402940 learned_addr learned my addr 172.21.15.161:0/206402940 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228359e640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2284161820 0x7f228415f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f227bfff640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d29d0 0x7f22840c79e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 -- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d29d0 msgr2=0x7f22840c79e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d29d0 0x7f22840c79e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 -- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2284161820 msgr2=0x7f228415f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2284161820 0x7f228415f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 -- 172.21.15.161:0/206402940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f227c009b30 con 0x7f22840d1270 2024-03-23T14:37:08.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228359e640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2284161820 0x7f228415f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f2282d9d640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d1270 0x7f22841599f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f227400c850 tx=0x7f227400cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f227b7fe640 1 -- 172.21.15.161:0/206402940 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274007830 con 0x7f22840d1270 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f227b7fe640 1 -- 172.21.15.161:0/206402940 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2274007990 con 0x7f22840d1270 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22840ca340 con 0x7f22840d1270 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f227b7fe640 1 -- 172.21.15.161:0/206402940 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2274007d00 con 0x7f22840d1270 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.695+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22840c7f20 con 0x7f22840d1270 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.697+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22840ca860 con 0x7f22840d1270 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.697+0000 7f227b7fe640 1 -- 172.21.15.161:0/206402940 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f22740113f0 con 0x7f22840d1270 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.697+0000 7f227b7fe640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f226406b810 0x7f226406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.698+0000 7f227b7fe640 1 -- 172.21.15.161:0/206402940 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f227408a310 con 0x7f22840d1270 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.698+0000 7f227bfff640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f226406b810 0x7f226406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:08.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.698+0000 7f227bfff640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f226406b810 0x7f226406dcd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f22840a1dd0 tx=0x7f226c008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:08.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.702+0000 7f227b7fe640 1 -- 172.21.15.161:0/206402940 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f22840ca860 con 0x7f22840d1270 2024-03-23T14:37:08.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.879+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f22840b6c90 con 0x7f22840d1270 2024-03-23T14:37:08.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.879+0000 7f227b7fe640 1 -- 172.21.15.161:0/206402940 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7f227401f020 con 0x7f22840d1270 2024-03-23T14:37:08.875 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:37:08.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f226406b810 msgr2=0x7f226406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f226406b810 0x7f226406dcd0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f22840a1dd0 tx=0x7f226c008040 comp rx=0 tx=0).stop 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d1270 msgr2=0x7f22841599f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d1270 0x7f22841599f0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7f227400c850 tx=0x7f227400cd10 comp rx=0 tx=0).stop 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 shutdown_connections 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f226406b810 0x7f226406dcd0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2284161820 0x7f228415f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22840d29d0 0x7f22840c79e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 --2- 172.21.15.161:0/206402940 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22840d1270 0x7f22841599f0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 >> 172.21.15.161:0/206402940 conn(0x7f228409d190 msgr2=0x7f22840cff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 shutdown_connections 2024-03-23T14:37:08.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:08.881+0000 7f228916c640 1 -- 172.21.15.161:0/206402940 wait complete. 2024-03-23T14:37:08.892 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:37:08.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.003+0000 7f9f81d97640 1 Processor -- start 2024-03-23T14:37:08.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.003+0000 7f9f81d97640 1 -- start start 2024-03-23T14:37:09.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f81d97640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9f7c06eca0 0x7f9f7c165d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:09.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f81d97640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f7c15be70 con 0x7f9f7c0724e0 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f81d97640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f7c15c150 con 0x7f9f7c072100 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f81d97640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f9f7c15c430 con 0x7f9f7c06eca0 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f7affd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9f7c0724e0 0x7f9f7c06e590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42962/0 (socket says 172.21.15.161:42962) 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f7affd640 1 -- 172.21.15.161:0/462218508 learned_addr learned my addr 172.21.15.161:0/462218508 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877470524 0 0) 0x7f9f7c15be70 con 0x7f9f7c0724e0 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f7bfff640 1 --2- 172.21.15.161:0/462218508 >> v2:172.21.15.183:3300/0 conn(0x7f9f7c06eca0 0x7f9f7c165d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.004+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9f64003520 con 0x7f9f7c0724e0 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 691398144 0 0) 0x7f9f64003520 con 0x7f9f7c0724e0 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 >> v2:172.21.15.183:3300/0 conn(0x7f9f7c06eca0 msgr2=0x7f9f7c165d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 --2- 172.21.15.161:0/462218508 >> v2:172.21.15.183:3300/0 conn(0x7f9f7c06eca0 0x7f9f7c165d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 >> v1:172.21.15.183:6789/0 conn(0x7f9f7c072100 legacy=0x7f9f7c15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:09.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7c0b2c70 con 0x7f9f7c0724e0 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f9f70001690 con 0x7f9f7c0724e0 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 >> v1:172.21.15.161:6789/0 conn(0x7f9f7c0724e0 legacy=0x7f9f7c06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f64003de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f640047c0 0x7f9f64004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f640051d0 0x7f9f6400b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c15c430 con 0x7f9f640039c0 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9f6400c120 con 0x7f9f640051d0 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9f6400c2e0 con 0x7f9f640047c0 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7b7fe640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f64003de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7bfff640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f640051d0 0x7f9f6400b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7affd640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f640047c0 0x7f9f64004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7b7fe640 1 -- 172.21.15.161:0/462218508 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f640047c0 msgr2=0x7f9f64004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7b7fe640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f640047c0 0x7f9f64004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7b7fe640 1 -- 172.21.15.161:0/462218508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f640051d0 msgr2=0x7f9f6400b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:09.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7b7fe640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f640051d0 0x7f9f6400b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7b7fe640 1 -- 172.21.15.161:0/462218508 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f64003520 con 0x7f9f640039c0 2024-03-23T14:37:09.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.005+0000 7f9f7bfff640 1 -- 172.21.15.161:0/462218508 reap_dead start 2024-03-23T14:37:09.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.006+0000 7f9f7b7fe640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f64003de0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c007fd0 tx=0x7f9f640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f4cc84fd6b43f7a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:09.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.006+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0052d0 con 0x7f9f640039c0 2024-03-23T14:37:09.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.006+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f6c005430 con 0x7f9f640039c0 2024-03-23T14:37:09.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.006+0000 7f9f7a7fc640 1 -- 172.21.15.161:0/462218508 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c02ea10 con 0x7f9f640039c0 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.006+0000 7f9f81d97640 1 -- 172.21.15.161:0/462218508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 msgr2=0x7f9f64003de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.006+0000 7f9f81d97640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f64003de0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c007fd0 tx=0x7f9f640057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.007+0000 7f9f81d97640 1 -- 172.21.15.161:0/462218508 shutdown_connections 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.007+0000 7f9f81d97640 1 --2- 172.21.15.161:0/462218508 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f64003de0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.007+0000 7f9f81d97640 1 -- 172.21.15.161:0/462218508 >> 172.21.15.161:0/462218508 conn(0x7f9f7c0b17f0 msgr2=0x7f9f7c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.007+0000 7f9f81d97640 1 -- 172.21.15.161:0/462218508 shutdown_connections 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.007+0000 7f9f81d97640 1 -- 172.21.15.161:0/462218508 wait complete. 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.008+0000 7f9f81d97640 1 Processor -- start 2024-03-23T14:37:09.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.008+0000 7f9f81d97640 1 -- start start 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f81d97640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f7c0aa4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f81d97640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f7c15a130 0x7f9f7c0ab150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f81d97640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f7c15a5b0 0x7f9f7c0ab690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f81d97640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c0a94a0 con 0x7f9f640039c0 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f81d97640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9f7c0a96f0 con 0x7f9f7c15a5b0 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f81d97640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9f7c0a99d0 con 0x7f9f7c15a130 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f7c0aa4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f7c0aa4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:45342/0 (socket says 172.21.15.161:45342) 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 -- 172.21.15.161:0/3949175947 learned_addr learned my addr 172.21.15.161:0/3949175947 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7affd640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f7c15a130 0x7f9f7c0ab150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7bfff640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f7c15a5b0 0x7f9f7c0ab690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 -- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f7c15a130 msgr2=0x7f9f7c0ab150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f7c15a130 0x7f9f7c0ab150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 -- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f7c15a5b0 msgr2=0x7f9f7c0ab690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:09.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f7c15a5b0 0x7f9f7c0ab690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 -- 172.21.15.161:0/3949175947 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f7c0a9cb0 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f7b7fe640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f7c0aa4b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c000c00 tx=0x7f9f6c001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f63fff640 1 -- 172.21.15.161:0/3949175947 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c007a30 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f63fff640 1 -- 172.21.15.161:0/3949175947 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f6c007240 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f7c0a9f90 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f63fff640 1 -- 172.21.15.161:0/3949175947 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f6c0075b0 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.009+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f7c1f7890 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.011+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9f7c153880 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.011+0000 7f9f63fff640 1 -- 172.21.15.161:0/3949175947 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9f6c0067d0 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.011+0000 7f9f63fff640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f6806b810 0x7f9f6806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.011+0000 7f9f63fff640 1 -- 172.21.15.161:0/3949175947 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f9f6c0ab6b0 con 0x7f9f640039c0 2024-03-23T14:37:09.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.011+0000 7f9f7affd640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f6806b810 0x7f9f6806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:09.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.012+0000 7f9f7affd640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f6806b810 0x7f9f6806dcd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0b1730 tx=0x7f9f70009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:09.011 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.015+0000 7f9f63fff640 1 -- 172.21.15.161:0/3949175947 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9f6c0760d0 con 0x7f9f640039c0 2024-03-23T14:37:09.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.204+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f9f7c153880 con 0x7f9f640039c0 2024-03-23T14:37:09.200 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.204+0000 7f9f63fff640 1 -- 172.21.15.161:0/3949175947 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) ==== 80+0+0 (secure 0 0 0) 0x7f9f6c079790 con 0x7f9f640039c0 2024-03-23T14:37:09.200 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f6806b810 msgr2=0x7f9f6806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f6806b810 0x7f9f6806dcd0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0b1730 tx=0x7f9f70009040 comp rx=0 tx=0).stop 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 msgr2=0x7f9f7c0aa4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f7c0aa4b0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f9f6c000c00 tx=0x7f9f6c001120 comp rx=0 tx=0).stop 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 shutdown_connections 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9f6806b810 0x7f9f6806dcd0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9f7c15a5b0 0x7f9f7c0ab690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9f7c15a130 0x7f9f7c0ab150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 --2- 172.21.15.161:0/3949175947 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9f640039c0 0x7f9f7c0aa4b0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.206+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 >> 172.21.15.161:0/3949175947 conn(0x7f9f7c0b17f0 msgr2=0x7f9f7c15c670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.207+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 shutdown_connections 2024-03-23T14:37:09.203 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:09.207+0000 7f9f81d97640 1 -- 172.21.15.161:0/3949175947 wait complete. 2024-03-23T14:37:09.213 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-23T14:37:09.213 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:09.213 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:09.239 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[56996]: fuse finished with error 0 and tester_r 0 2024-03-23T14:37:09.240 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:37:15.342 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:37:15.343 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:15.343 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T14:37:15.358 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:37:15.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.504+0000 7feb82e88640 1 Processor -- start 2024-03-23T14:37:15.500 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.504+0000 7feb82e88640 1 -- start start 2024-03-23T14:37:15.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.505+0000 7feb82e88640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7feb7c15bd50 0x7feb7c15c130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.505+0000 7feb82e88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c0d18a0 0x7feb7c0d1d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.505+0000 7feb82e88640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb7c0d2260 con 0x7feb7c0cf8a0 2024-03-23T14:37:15.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.505+0000 7feb82e88640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7feb7c0d0b90 con 0x7feb7c15bd50 2024-03-23T14:37:15.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.505+0000 7feb82e88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feb7c0d05d0 con 0x7feb7c0d18a0 2024-03-23T14:37:15.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.505+0000 7feb81425640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7feb7c0cf8a0 0x7feb7c0cfd10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44478/0 (socket says 172.21.15.161:44478) 2024-03-23T14:37:15.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb81425640 1 -- 172.21.15.161:0/3751208965 learned_addr learned my addr 172.21.15.161:0/3751208965 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7bfff640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c0d18a0 0x7feb7c0d1d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb80c24640 1 --2- 172.21.15.161:0/3751208965 >> v2:172.21.15.183:3300/0 conn(0x7feb7c15bd50 0x7feb7c15c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 472604780 0 0) 0x7feb7c0d2260 con 0x7feb7c0cf8a0 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb6c003520 con 0x7feb7c0cf8a0 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3110283953 0 0) 0x7feb6c003520 con 0x7feb7c0cf8a0 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c0d18a0 msgr2=0x7feb7c0d1d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c0d18a0 0x7feb7c0d1d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 >> v2:172.21.15.183:3300/0 conn(0x7feb7c15bd50 msgr2=0x7feb7c15c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 --2- 172.21.15.161:0/3751208965 >> v2:172.21.15.183:3300/0 conn(0x7feb7c15bd50 0x7feb7c15c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c0d08b0 con 0x7feb7c0cf8a0 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7feb740019f0 con 0x7feb7c0cf8a0 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 >> v1:172.21.15.161:6789/0 conn(0x7feb7c0cf8a0 legacy=0x7feb7c0cfd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c0d18a0 0x7feb6c003970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c003ef0 0x7feb6c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb6c004d00 0x7feb6c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c0d0b90 con 0x7feb6c004d00 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c0d05d0 con 0x7feb6c003ef0 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feb6c00ba60 con 0x7feb7c0d18a0 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb81425640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb6c004d00 0x7feb6c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb80c24640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c0d18a0 0x7feb6c003970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.506+0000 7feb7bfff640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c003ef0 0x7feb6c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb81425640 1 -- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c0d18a0 msgr2=0x7feb6c003970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb81425640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c0d18a0 0x7feb6c003970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb81425640 1 -- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c003ef0 msgr2=0x7feb6c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb81425640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c003ef0 0x7feb6c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb81425640 1 -- 172.21.15.161:0/3751208965 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6c003520 con 0x7feb6c004d00 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb7bfff640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb6c003ef0 0x7feb6c0047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb81425640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb6c004d00 0x7feb6c005180 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7feb7400a8d0 tx=0x7feb6c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c5630300bbe126 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb74034630 con 0x7feb6c004d00 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb7402f070 con 0x7feb6c004d00 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.507+0000 7feb7b7fe640 1 -- 172.21.15.161:0/3751208965 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb740349d0 con 0x7feb6c004d00 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.508+0000 7feb82e88640 1 -- 172.21.15.161:0/3751208965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb6c004d00 msgr2=0x7feb6c005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.508+0000 7feb82e88640 1 --2- 172.21.15.161:0/3751208965 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb6c004d00 0x7feb6c005180 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7feb7400a8d0 tx=0x7feb6c0057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.508+0000 7feb81425640 1 -- 172.21.15.161:0/3751208965 reap_dead start 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.508+0000 7feb82e88640 1 -- 172.21.15.161:0/3751208965 shutdown_connections 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.508+0000 7feb82e88640 1 -- 172.21.15.161:0/3751208965 >> 172.21.15.161:0/3751208965 conn(0x7feb7c09aa40 msgr2=0x7feb7c09ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.508+0000 7feb82e88640 1 -- 172.21.15.161:0/3751208965 shutdown_connections 2024-03-23T14:37:15.507 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.508+0000 7feb82e88640 1 -- 172.21.15.161:0/3751208965 wait complete. 2024-03-23T14:37:15.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.509+0000 7feb82e88640 1 Processor -- start 2024-03-23T14:37:15.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.509+0000 7feb82e88640 1 -- start start 2024-03-23T14:37:15.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb82e88640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb7c159500 0x7feb7c0c92f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb82e88640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c159bb0 0x7feb7c0ce980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb82e88640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb7c15a100 0x7feb7c0ceec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.510 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb82e88640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c0c9830 con 0x7feb7c159500 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb82e88640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feb7c0b4880 con 0x7feb7c15a100 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb82e88640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feb7c0b56e0 con 0x7feb7c159bb0 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb80c24640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb7c159500 0x7feb7c0c92f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb80c24640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb7c159500 0x7feb7c0c92f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37758/0 (socket says 172.21.15.161:37758) 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb80c24640 1 -- 172.21.15.161:0/2809045153 learned_addr learned my addr 172.21.15.161:0/2809045153 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb81425640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb7c15a100 0x7feb7c0ceec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb7bfff640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c159bb0 0x7feb7c0ce980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb80c24640 1 -- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c159bb0 msgr2=0x7feb7c0ce980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb80c24640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c159bb0 0x7feb7c0ce980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb80c24640 1 -- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb7c15a100 msgr2=0x7feb7c0ceec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:15.511 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb80c24640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb7c15a100 0x7feb7c0ceec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.510+0000 7feb80c24640 1 -- 172.21.15.161:0/2809045153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7c0b4ad0 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.511+0000 7feb80c24640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb7c159500 0x7feb7c0c92f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7feb70011a70 tx=0x7feb70011f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.511+0000 7feb797fa640 1 -- 172.21.15.161:0/2809045153 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb70009680 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.511+0000 7feb797fa640 1 -- 172.21.15.161:0/2809045153 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb7000fb50 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.511+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb7c0b4db0 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.511+0000 7feb797fa640 1 -- 172.21.15.161:0/2809045153 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb7000fee0 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.511+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb7c0b5370 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.512+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb7c068bb0 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.512+0000 7feb797fa640 1 -- 172.21.15.161:0/2809045153 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feb70014440 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.513+0000 7feb797fa640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb6006b810 0x7feb6006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.513+0000 7feb797fa640 1 -- 172.21.15.161:0/2809045153 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7feb7008aa80 con 0x7feb7c159500 2024-03-23T14:37:15.512 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.513+0000 7feb7bfff640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb6006b810 0x7feb6006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:15.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.513+0000 7feb7bfff640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb6006b810 0x7feb6006dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feb7c1f7740 tx=0x7feb68008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:15.513 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.517+0000 7feb797fa640 1 -- 172.21.15.161:0/2809045153 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feb70055530 con 0x7feb7c159500 2024-03-23T14:37:15.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.733+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7feb7c09fcf0 con 0x7feb7c159500 2024-03-23T14:37:15.729 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.733+0000 7feb797fa640 1 -- 172.21.15.161:0/2809045153 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7feb70058bf0 con 0x7feb7c159500 2024-03-23T14:37:15.729 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T14:37:15.731 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb6006b810 msgr2=0x7feb6006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb6006b810 0x7feb6006dcd0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7feb7c1f7740 tx=0x7feb68008040 comp rx=0 tx=0).stop 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb7c159500 msgr2=0x7feb7c0c92f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb7c159500 0x7feb7c0c92f0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7feb70011a70 tx=0x7feb70011f30 comp rx=0 tx=0).stop 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 shutdown_connections 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb6006b810 0x7feb6006dcd0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feb7c15a100 0x7feb7c0ceec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feb7c159bb0 0x7feb7c0ce980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 --2- 172.21.15.161:0/2809045153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feb7c159500 0x7feb7c0c92f0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.735+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 >> 172.21.15.161:0/2809045153 conn(0x7feb7c09aa40 msgr2=0x7feb7c0cdc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.736+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 shutdown_connections 2024-03-23T14:37:15.732 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15.736+0000 7feb82e88640 1 -- 172.21.15.161:0/2809045153 wait complete. 2024-03-23T14:37:15.742 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:37:15.742 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:37:15.742 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-23T14:37:15.742 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T14:37:15.742 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:15.742 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:37:15.743 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:37:15.743 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:37:15.743 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:15.743 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:15.743 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-23T14:37:15.758 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:37:15.758 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:37:15.758 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:37:15.758 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:15.758 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:37:15.758 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:15.758 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42634sec preferred_lft 42634sec 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:15.759 INFO:teuthology.orchestra.run.smithi161.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c4a9:64ff:fe59:877a/64 scope link 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c48:50ff:fe01:9f4e/64 scope link 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 62:02:d5:17:e7:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::6002:d5ff:fe17:e70d/64 scope link 2024-03-23T14:37:15.760 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:15.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:15.761 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:37:15.824 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-23T14:37:15.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:15.826 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-23T14:37:15.881 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-23T14:37:15.881 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:37:15.881 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:37:15.881 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:37:15.881 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:37:15.882 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:37:15.882 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:37:15.882 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:37:15.882 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:37:15.882 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:37:15.882 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:37:15.969 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:37:16.010 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:37:16.017 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:16.017 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:37:16.031 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T14:37:16.032 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T14:37:16.032 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:16.032 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:16.032 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-23T14:37:16.098 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T14:37:16.099 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:37:16.099 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:16.153 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:16.154 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:37:16.154 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:16.208 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:37:16.276 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:37:16.332 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:37:16.332 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:37:16.332 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:37:16.332 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:37:16.332 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:37:16.333 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:16.334 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:16.335 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:16.335 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:37:16.387 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T14:37:16.387 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T14:37:16.429 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:37:16.455 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:37:16.511 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:37:16.511 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:37:16.511 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:37:16.511 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:37:16.511 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:37:16.512 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:37:16.513 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:16.514 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:16.514 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:16.514 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:16.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:16.514 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:37:16.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:2024-03-23T14:37:16.520+0000 7f6d9788f480 -1 init, newargv = 0x55a70eb14eb0 newargc=15 2024-03-23T14:37:16.515 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[57973]: starting ceph client 2024-03-23T14:37:16.520 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[57973]: starting fuse 2024-03-23T14:37:16.530 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:37:16.531 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-23T14:37:16.531 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:37:16.531 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-23T14:37:16.531 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:37:16.532 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:37:16.533 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:37:16.533 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-23T14:37:16.533 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-23T14:37:16.533 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-23T14:37:16.533 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:37:16.533 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:16.533 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.0")) 2024-03-23T14:37:16.533 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:37:16.651 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.0.57973.asok 2024-03-23T14:37:16.654 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:37:16.661 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.57973.asok 2024-03-23T14:37:16.662 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:16.662 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.57973.asok status 2024-03-23T14:37:16.798 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:16.798 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-23T14:37:16.798 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:37:16.798 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:37:16.798 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "0", 2024-03-23T14:37:16.798 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-23T14:37:16.798 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "57973", 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6168, 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 6168 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:37:16.799 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3770117117 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 3770117117 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.6168 192.168.0.1:0/3770117117", 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/3770117117", 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 23, 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 48, 2024-03-23T14:37:16.800 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:37:16.801 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-23T14:37:16.801 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-23T14:37:16.801 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:16.806 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:16.807 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-23T14:37:16.807 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:16.822 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-23T14:37:16.822 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:16.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:16.822 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:16.890 DEBUG:teuthology.orchestra.run.smithi161:> stdin-killer -- python3 -c ' 2024-03-23T14:37:16.890 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:37:16.890 DEBUG:teuthology.orchestra.run.smithi161:> import errno 2024-03-23T14:37:16.890 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:16.890 DEBUG:teuthology.orchestra.run.smithi161:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> try: 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> if False: 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> ret = os.read(fd, 1024) 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> else: 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> os.write(fd, b'"'"'content'"'"') 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> except OSError as e: 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> if e.errno != errno.EPERM: 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> raise 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> else: 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("client does not check permission of data pool") 2024-03-23T14:37:16.891 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:37:17.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:37:17.007 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow w pool=cephfs_data' 2024-03-23T14:37:17.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.160+0000 7fae38215640 1 Processor -- start 2024-03-23T14:37:17.156 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.161+0000 7fae38215640 1 -- start start 2024-03-23T14:37:17.158 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae38215640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fae3015e440 0x7fae300d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae38215640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae300d1d50 0x7fae300d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae38215640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fae300d2a20 0x7fae300ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae38215640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fae300d2eb0 con 0x7fae300d2a20 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae38215640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fae300b0ac0 con 0x7fae3015e440 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae38215640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fae300b4220 con 0x7fae300d1d50 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae367b2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fae300d2a20 0x7fae300ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae367b2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fae300d2a20 0x7fae300ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37762/0 (socket says 172.21.15.161:37762) 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae367b2640 1 -- 172.21.15.161:0/763284894 learned_addr learned my addr 172.21.15.161:0/763284894 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae35fb1640 1 --2- 172.21.15.161:0/763284894 >> v2:172.21.15.183:3300/0 conn(0x7fae3015e440 0x7fae300d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae357b0640 1 --2- 172.21.15.161:0/763284894 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae300d1d50 0x7fae300d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae367b2640 1 -- 172.21.15.161:0/763284894 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae300d1d50 msgr2=0x7fae300d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae367b2640 1 --2- 172.21.15.161:0/763284894 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae300d1d50 0x7fae300d24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae367b2640 1 -- 172.21.15.161:0/763284894 >> v2:172.21.15.183:3300/0 conn(0x7fae3015e440 msgr2=0x7fae300d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:17.159 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae367b2640 1 --2- 172.21.15.161:0/763284894 >> v2:172.21.15.183:3300/0 conn(0x7fae3015e440 0x7fae300d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.160 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.162+0000 7fae367b2640 1 -- 172.21.15.161:0/763284894 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae300b3350 con 0x7fae300d2a20 2024-03-23T14:37:17.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.163+0000 7fae367b2640 1 --2- 172.21.15.161:0/763284894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300ceaa0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fae3008e5c0 tx=0x7fae2c02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3183b125b3b81b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.163+0000 7fae34faf640 1 -- 172.21.15.161:0/763284894 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c02ca70 con 0x7fae300d2a20 2024-03-23T14:37:17.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.163+0000 7fae34faf640 1 -- 172.21.15.161:0/763284894 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae2c02cbd0 con 0x7fae300d2a20 2024-03-23T14:37:17.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.163+0000 7fae34faf640 1 -- 172.21.15.161:0/763284894 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2c0343f0 con 0x7fae300d2a20 2024-03-23T14:37:17.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 -- 172.21.15.161:0/763284894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 msgr2=0x7fae300ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 --2- 172.21.15.161:0/763284894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300ceaa0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7fae3008e5c0 tx=0x7fae2c02bed0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 -- 172.21.15.161:0/763284894 shutdown_connections 2024-03-23T14:37:17.161 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 --2- 172.21.15.161:0/763284894 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300ceaa0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 --2- 172.21.15.161:0/763284894 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae300d1d50 0x7fae300d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 --2- 172.21.15.161:0/763284894 >> v2:172.21.15.183:3300/0 conn(0x7fae3015e440 0x7fae300d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 -- 172.21.15.161:0/763284894 >> 172.21.15.161:0/763284894 conn(0x7fae300d3f00 msgr2=0x7fae300d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:17.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 -- 172.21.15.161:0/763284894 shutdown_connections 2024-03-23T14:37:17.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.164+0000 7fae38215640 1 -- 172.21.15.161:0/763284894 wait complete. 2024-03-23T14:37:17.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.165+0000 7fae38215640 1 Processor -- start 2024-03-23T14:37:17.162 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.165+0000 7fae38215640 1 -- start start 2024-03-23T14:37:17.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae38215640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae300d1d50 0x7fae300cbfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae38215640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300cc520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae38215640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3015e440 0x7fae300cca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae38215640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fae3014f260 con 0x7fae300d2a20 2024-03-23T14:37:17.164 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae38215640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fae301f3b00 con 0x7fae300d1d50 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae38215640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fae300cafd0 con 0x7fae3015e440 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300cc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300cc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37770/0 (socket says 172.21.15.161:37770) 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 -- 172.21.15.161:0/3080785629 learned_addr learned my addr 172.21.15.161:0/3080785629 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 -- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3015e440 msgr2=0x7fae300cca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae367b2640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3015e440 0x7fae300cca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae35fb1640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae300d1d50 0x7fae300cbfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3015e440 0x7fae300cca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 -- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae300d1d50 msgr2=0x7fae300cbfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae300d1d50 0x7fae300cbfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 -- 172.21.15.161:0/3080785629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae2c009b30 con 0x7fae300d2a20 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.166+0000 7fae357b0640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300cc520 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fae2000e870 tx=0x7fae2000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.167+0000 7fae267fc640 1 -- 172.21.15.161:0/3080785629 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2000ee60 con 0x7fae300d2a20 2024-03-23T14:37:17.165 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.167+0000 7fae267fc640 1 -- 172.21.15.161:0/3080785629 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae2000c940 con 0x7fae300d2a20 2024-03-23T14:37:17.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.167+0000 7fae267fc640 1 -- 172.21.15.161:0/3080785629 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae2000ccf0 con 0x7fae300d2a20 2024-03-23T14:37:17.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.167+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae300cb220 con 0x7fae300d2a20 2024-03-23T14:37:17.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.167+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae300cb5b0 con 0x7fae300d2a20 2024-03-23T14:37:17.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.168+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fae300b4390 con 0x7fae300d2a20 2024-03-23T14:37:17.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.168+0000 7fae267fc640 1 -- 172.21.15.161:0/3080785629 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fae20013470 con 0x7fae300d2a20 2024-03-23T14:37:17.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.169+0000 7fae267fc640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fae1406b810 0x7fae1406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.169+0000 7fae267fc640 1 -- 172.21.15.161:0/3080785629 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fae2008a110 con 0x7fae300d2a20 2024-03-23T14:37:17.166 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.169+0000 7fae35fb1640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fae1406b810 0x7fae1406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.167 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.169+0000 7fae35fb1640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fae1406b810 0x7fae1406dcd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fae28009970 tx=0x7fae28008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.168 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.173+0000 7fae267fc640 1 -- 172.21.15.161:0/3080785629 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fae20054bc0 con 0x7fae300d2a20 2024-03-23T14:37:17.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.420+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]} v 0) -- 0x7fae30068160 con 0x7fae300d2a20 2024-03-23T14:37:17.422 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.425+0000 7fae267fc640 1 -- 172.21.15.161:0/3080785629 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow w pool=cephfs_data"]}]=0 updated caps for client.0 v17) ==== 183+0+140 (secure 0 0 0) 0x7fae20058280 con 0x7fae300d2a20 2024-03-23T14:37:17.422 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:37:17.422 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:37:17.423 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-23T14:37:17.423 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-23T14:37:17.423 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow w pool=cephfs_data" 2024-03-23T14:37:17.423 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-23T14:37:17.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fae1406b810 msgr2=0x7fae1406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fae1406b810 0x7fae1406dcd0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fae28009970 tx=0x7fae28008040 comp rx=0 tx=0).stop 2024-03-23T14:37:17.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 msgr2=0x7fae300cc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300cc520 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7fae2000e870 tx=0x7fae2000ed30 comp rx=0 tx=0).stop 2024-03-23T14:37:17.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 shutdown_connections 2024-03-23T14:37:17.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fae1406b810 0x7fae1406dcd0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fae3015e440 0x7fae300cca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.425 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fae300d2a20 0x7fae300cc520 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 --2- 172.21.15.161:0/3080785629 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fae300d1d50 0x7fae300cbfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 >> 172.21.15.161:0/3080785629 conn(0x7fae300d3f00 msgr2=0x7fae300cf3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:17.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 shutdown_connections 2024-03-23T14:37:17.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.428+0000 7fae38215640 1 -- 172.21.15.161:0/3080785629 wait complete. 2024-03-23T14:37:17.435 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:37:17.435 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:17.451 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.596+0000 7f1416c8f640 1 Processor -- start 2024-03-23T14:37:17.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.596+0000 7f1416c8f640 1 -- start start 2024-03-23T14:37:17.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f1416c8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f14100b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f1416c8f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f14100b3610 0x7f14100b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f1416c8f640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1410070b90 con 0x7f14100b4260 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f1416c8f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f14100b3fe0 con 0x7f14100b3610 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f1416c8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f141015ab90 con 0x7f141006eb30 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f141522c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f14100b3610 0x7f14100b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f140ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f14100b4260 0x7f141015b560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44482/0 (socket says 172.21.15.161:44482) 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f140ffff640 1 -- 172.21.15.161:0/2073847659 learned_addr learned my addr 172.21.15.161:0/2073847659 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f141522c640 1 -- 172.21.15.161:0/2073847659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 msgr2=0x7f14100b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f1414a2b640 1 --2- 172.21.15.161:0/2073847659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f14100b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f141522c640 1 --2- 172.21.15.161:0/2073847659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f14100b4870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f141522c640 1 -- 172.21.15.161:0/2073847659 >> v1:172.21.15.161:6789/0 conn(0x7f14100b4260 legacy=0x7f141015b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f140f7fe640 1 -- 172.21.15.161:0/2073847659 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1065889482 0 0) 0x7f1410070b90 con 0x7f14100b4260 2024-03-23T14:37:17.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f141522c640 1 -- 172.21.15.161:0/2073847659 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141015b0d0 con 0x7f14100b3610 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.597+0000 7f1414a2b640 1 --2- 172.21.15.161:0/2073847659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f14100b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.598+0000 7f141522c640 1 --2- 172.21.15.161:0/2073847659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f14100b3aa0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1408009d90 tx=0x7f140802cf20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07fa59eb6a9b3bb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.598+0000 7f140f7fe640 1 -- 172.21.15.161:0/2073847659 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140802dcd0 con 0x7f14100b3610 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.598+0000 7f140f7fe640 1 -- 172.21.15.161:0/2073847659 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f140802de30 con 0x7f14100b3610 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.598+0000 7f140f7fe640 1 -- 172.21.15.161:0/2073847659 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408035660 con 0x7f14100b3610 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.598+0000 7f1416c8f640 1 -- 172.21.15.161:0/2073847659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 msgr2=0x7f14100b3aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.598+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2073847659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f14100b3aa0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f1408009d90 tx=0x7f140802cf20 comp rx=0 tx=0).stop 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.599+0000 7f1416c8f640 1 -- 172.21.15.161:0/2073847659 shutdown_connections 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.599+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2073847659 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f14100b3aa0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.599+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2073847659 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f14100b4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.599+0000 7f1416c8f640 1 -- 172.21.15.161:0/2073847659 >> 172.21.15.161:0/2073847659 conn(0x7f14100b17f0 msgr2=0x7f14100b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.599+0000 7f1416c8f640 1 -- 172.21.15.161:0/2073847659 shutdown_connections 2024-03-23T14:37:17.597 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.599+0000 7f1416c8f640 1 -- 172.21.15.161:0/2073847659 wait complete. 2024-03-23T14:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.600+0000 7f1416c8f640 1 Processor -- start 2024-03-23T14:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.600+0000 7f1416c8f640 1 -- start start 2024-03-23T14:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1416c8f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f141014bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1416c8f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f141014ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1416c8f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14100b4260 0x7f141014e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1416c8f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f141014e390 con 0x7f14100b3610 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1416c8f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f141014d780 con 0x7f14100b4260 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1416c8f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f141014da60 con 0x7f141006eb30 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f141014ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f141014ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37782/0 (socket says 172.21.15.161:37782) 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 -- 172.21.15.161:0/2160491340 learned_addr learned my addr 172.21.15.161:0/2160491340 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1414a2b640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f141014bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f141522c640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14100b4260 0x7f141014e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 -- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 msgr2=0x7f141014bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f141014bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 -- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14100b4260 msgr2=0x7f141014e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14100b4260 0x7f141014e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 -- 172.21.15.161:0/2160491340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1408009a70 con 0x7f14100b3610 2024-03-23T14:37:17.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f141522c640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14100b4260 0x7f141014e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140ffff640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f141014ce10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00c430 tx=0x7f13fc00c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140d7fa640 1 -- 172.21.15.161:0/2160491340 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13fc0074e0 con 0x7f14100b3610 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140d7fa640 1 -- 172.21.15.161:0/2160491340 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13fc007660 con 0x7f14100b3610 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141014dd40 con 0x7f14100b3610 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f140d7fa640 1 -- 172.21.15.161:0/2160491340 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13fc004a20 con 0x7f14100b3610 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.601+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f141014c4b0 con 0x7f14100b3610 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.602+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141014e540 con 0x7f14100b3610 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.603+0000 7f140d7fa640 1 -- 172.21.15.161:0/2160491340 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f13fc004b80 con 0x7f14100b3610 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.603+0000 7f140d7fa640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13f406b810 0x7f13f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.603+0000 7f140d7fa640 1 -- 172.21.15.161:0/2160491340 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f13fc089fb0 con 0x7f14100b3610 2024-03-23T14:37:17.602 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.604+0000 7f1414a2b640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13f406b810 0x7f13f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.603 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.604+0000 7f1414a2b640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13f406b810 0x7f13f406dcd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f140400ab60 tx=0x7f1404008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.608+0000 7f140d7fa640 1 -- 172.21.15.161:0/2160491340 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f141014e540 con 0x7f14100b3610 2024-03-23T14:37:17.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.787+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1410071360 con 0x7f14100b3610 2024-03-23T14:37:17.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.788+0000 7f140d7fa640 1 -- 172.21.15.161:0/2160491340 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7f13fc054940 con 0x7f14100b3610 2024-03-23T14:37:17.784 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:17.784 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13f406b810 msgr2=0x7f13f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13f406b810 0x7f13f406dcd0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f140400ab60 tx=0x7f1404008040 comp rx=0 tx=0).stop 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 msgr2=0x7f141014ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f141014ce10 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f13fc00c430 tx=0x7f13fc00c8f0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 shutdown_connections 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13f406b810 0x7f13f406dcd0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f14100b4260 0x7f141014e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f14100b3610 0x7f141014ce10 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 --2- 172.21.15.161:0/2160491340 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f141006eb30 0x7f141014bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 >> 172.21.15.161:0/2160491340 conn(0x7f14100b17f0 msgr2=0x7f1410154600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:17.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 shutdown_connections 2024-03-23T14:37:17.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.789+0000 7f1416c8f640 1 -- 172.21.15.161:0/2160491340 wait complete. 2024-03-23T14:37:17.797 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:56.747667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:57.478793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:37:17.797 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.906+0000 7fc0a8209640 1 Processor -- start 2024-03-23T14:37:17.902 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.907+0000 7fc0a8209640 1 -- start start 2024-03-23T14:37:17.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a8209640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a00d1270 0x7fc0a00d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a8209640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc0a00d29d0 0x7fc0a00b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a8209640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0a0161c00 con 0x7fc0a0161820 2024-03-23T14:37:17.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a8209640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc0a00d2e20 con 0x7fc0a00d29d0 2024-03-23T14:37:17.907 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a8209640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc0a00cea30 con 0x7fc0a00d1270 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a5fa5640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc0a0161820 0x7fc0a00d47e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44492/0 (socket says 172.21.15.161:44492) 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a5fa5640 1 -- 172.21.15.161:0/3926937975 learned_addr learned my addr 172.21.15.161:0/3926937975 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a67a6640 1 --2- 172.21.15.161:0/3926937975 >> v2:172.21.15.183:3300/0 conn(0x7fc0a00d29d0 0x7fc0a00b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1750011612 0 0) 0x7fc0a0161c00 con 0x7fc0a0161820 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc08c003520 con 0x7fc0a0161820 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a57a4640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a00d1270 0x7fc0a00d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2659231697 0 0) 0x7fc08c003520 con 0x7fc0a0161820 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a00d1270 msgr2=0x7fc0a00d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a00d1270 0x7fc0a00d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 >> v2:172.21.15.183:3300/0 conn(0x7fc0a00d29d0 msgr2=0x7fc0a00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 --2- 172.21.15.161:0/3926937975 >> v2:172.21.15.183:3300/0 conn(0x7fc0a00d29d0 0x7fc0a00b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a00cfd70 con 0x7fc0a0161820 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.908+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fc09c001670 con 0x7fc0a0161820 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 >> v1:172.21.15.161:6789/0 conn(0x7fc0a0161820 legacy=0x7fc0a00d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a4fa3640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc08c003970 0x7fc08c003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a4fa3640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a00d1270 0x7fc08c0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a4fa3640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc08c004d00 0x7fc08c005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a00d2e20 con 0x7fc08c003970 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a00cea30 con 0x7fc08c004d00 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc08c00b980 con 0x7fc0a00d1270 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a5fa5640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc08c003970 0x7fc08c003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a67a6640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc08c004d00 0x7fc08c005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a57a4640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a00d1270 0x7fc08c0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a5fa5640 1 -- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a00d1270 msgr2=0x7fc08c0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:17.909 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a5fa5640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc0a00d1270 0x7fc08c0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a5fa5640 1 -- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc08c004d00 msgr2=0x7fc08c005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a5fa5640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc08c004d00 0x7fc08c005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.909+0000 7fc0a5fa5640 1 -- 172.21.15.161:0/3926937975 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08c003520 con 0x7fc08c003970 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a5fa5640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc08c003970 0x7fc08c003d90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00c290 tx=0x7fc08c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aeae2f4ba0900eb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c005ad0 con 0x7fc08c003970 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc09c005c30 con 0x7fc08c003970 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a4fa3640 1 -- 172.21.15.161:0/3926937975 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09c02c8c0 con 0x7fc08c003970 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a8209640 1 -- 172.21.15.161:0/3926937975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc08c003970 msgr2=0x7fc08c003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a8209640 1 --2- 172.21.15.161:0/3926937975 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc08c003970 0x7fc08c003d90 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7fc09c00c290 tx=0x7fc08c0057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a67a6640 1 -- 172.21.15.161:0/3926937975 reap_dead start 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a8209640 1 -- 172.21.15.161:0/3926937975 shutdown_connections 2024-03-23T14:37:17.910 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.910+0000 7fc0a8209640 1 -- 172.21.15.161:0/3926937975 >> 172.21.15.161:0/3926937975 conn(0x7fc0a009d190 msgr2=0x7fc0a00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:17.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.911+0000 7fc0a8209640 1 -- 172.21.15.161:0/3926937975 shutdown_connections 2024-03-23T14:37:17.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.911+0000 7fc0a8209640 1 -- 172.21.15.161:0/3926937975 wait complete. 2024-03-23T14:37:17.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.912+0000 7fc0a8209640 1 Processor -- start 2024-03-23T14:37:17.911 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.912+0000 7fc0a8209640 1 -- start start 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.912+0000 7fc0a8209640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc08c003970 0x7fc0a0153340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.912+0000 7fc0a8209640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a00b3d50 0x7fc0a00cb990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.912+0000 7fc0a8209640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a00b41d0 0x7fc0a00cc810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.912+0000 7fc0a8209640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a0153880 con 0x7fc0a00b3d50 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.912+0000 7fc0a8209640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc0a00b7a40 con 0x7fc0a00b41d0 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.912+0000 7fc0a8209640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc0a00b7d20 con 0x7fc08c003970 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a00b3d50 0x7fc0a00cb990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a00b3d50 0x7fc0a00cb990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37800/0 (socket says 172.21.15.161:37800) 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 -- 172.21.15.161:0/3281629459 learned_addr learned my addr 172.21.15.161:0/3281629459 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:17.914 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a5fa5640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc08c003970 0x7fc0a0153340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a67a6640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a00b41d0 0x7fc0a00cc810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 -- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc08c003970 msgr2=0x7fc0a0153340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc08c003970 0x7fc0a0153340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 -- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a00b41d0 msgr2=0x7fc0a00cc810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a00b41d0 0x7fc0a00cc810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 -- 172.21.15.161:0/3281629459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0a00b8000 con 0x7fc0a00b3d50 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a57a4640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a00b3d50 0x7fc0a00cb990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc0900118e0 tx=0x7fc090011da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0967fc640 1 -- 172.21.15.161:0/3281629459 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc09000faf0 con 0x7fc0a00b3d50 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0967fc640 1 -- 172.21.15.161:0/3281629459 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc09000fc50 con 0x7fc0a00b3d50 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0a00b82e0 con 0x7fc0a00b3d50 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0967fc640 1 -- 172.21.15.161:0/3281629459 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc090014440 con 0x7fc0a00b3d50 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.913+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0a00b88a0 con 0x7fc0a00b3d50 2024-03-23T14:37:17.915 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.915+0000 7fc0967fc640 1 -- 172.21.15.161:0/3281629459 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc0900145a0 con 0x7fc0a00b3d50 2024-03-23T14:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.915+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0a004f4e0 con 0x7fc0a00b3d50 2024-03-23T14:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.915+0000 7fc0967fc640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc08406b810 0x7fc08406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.915+0000 7fc0967fc640 1 -- 172.21.15.161:0/3281629459 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fc09008aa00 con 0x7fc0a00b3d50 2024-03-23T14:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.915+0000 7fc0a5fa5640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc08406b810 0x7fc08406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.916+0000 7fc0a5fa5640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc08406b810 0x7fc08406dcd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc09c034000 tx=0x7fc09c007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:17.916 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:17.919+0000 7fc0967fc640 1 -- 172.21.15.161:0/3281629459 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc090055470 con 0x7fc0a00b3d50 2024-03-23T14:37:18.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.096+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fc0a00d2270 con 0x7fc0a00b3d50 2024-03-23T14:37:18.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.096+0000 7fc0967fc640 1 -- 172.21.15.161:0/3281629459 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7fc090058b30 con 0x7fc0a00b3d50 2024-03-23T14:37:18.092 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:37:18.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc08406b810 msgr2=0x7fc08406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc08406b810 0x7fc08406dcd0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc09c034000 tx=0x7fc09c007000 comp rx=0 tx=0).stop 2024-03-23T14:37:18.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a00b3d50 msgr2=0x7fc0a00cb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a00b3d50 0x7fc0a00cb990 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7fc0900118e0 tx=0x7fc090011da0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 shutdown_connections 2024-03-23T14:37:18.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc08406b810 0x7fc08406dcd0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc0a00b41d0 0x7fc0a00cc810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc0a00b3d50 0x7fc0a00cb990 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 --2- 172.21.15.161:0/3281629459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc08c003970 0x7fc0a0153340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 >> 172.21.15.161:0/3281629459 conn(0x7fc0a009d190 msgr2=0x7fc0a0162350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:18.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 shutdown_connections 2024-03-23T14:37:18.095 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.098+0000 7fc0a8209640 1 -- 172.21.15.161:0/3281629459 wait complete. 2024-03-23T14:37:18.105 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:37:18.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.215+0000 7f43ba784640 1 Processor -- start 2024-03-23T14:37:18.211 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.216+0000 7f43ba784640 1 -- start start 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43ba784640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f43b415f560 0x7f43b40b1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43ba784640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43b40b0e30 con 0x7f43b402c3c0 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43ba784640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f43b40735e0 con 0x7f43b40afd20 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43ba784640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f43b415f940 con 0x7f43b415f560 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b37fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f43b402c3c0 0x7f43b40727c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44498/0 (socket says 172.21.15.161:44498) 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b37fe640 1 -- 172.21.15.161:0/3378983160 learned_addr learned my addr 172.21.15.161:0/3378983160 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f43b415f560 0x7f43b40b1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3732649666 0 0) 0x7f43b40b0e30 con 0x7f43b402c3c0 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f439c003520 con 0x7f43b402c3c0 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1838118216 0 0) 0x7f43b40735e0 con 0x7f43b40afd20 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f43b40b0e30 con 0x7f43b40afd20 2024-03-23T14:37:18.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1620818155 0 0) 0x7f439c003520 con 0x7f43b402c3c0 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f43b415f560 msgr2=0x7f43b40b1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f43b415f560 0x7f43b40b1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 >> v1:172.21.15.183:6789/0 conn(0x7f43b40afd20 legacy=0x7f43b4072ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b40b0220 con 0x7f43b402c3c0 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f43a8001670 con 0x7f43b402c3c0 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.217+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 >> v1:172.21.15.161:6789/0 conn(0x7f43b402c3c0 legacy=0x7f43b40727c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b2ffd640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f439c003b30 0x7f439c003f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b2ffd640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f439c0047c0 0x7f439c004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b2ffd640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c005180 0x7f439c00b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43b415f940 con 0x7f439c003b30 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f439c00bff0 con 0x7f439c0047c0 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f439c00c1b0 con 0x7f439c005180 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f43b415f560 0x7f43b40b1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f439c003b30 0x7f439c003f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.217 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b37fe640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f439c0047c0 0x7f439c004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b8d21640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c005180 0x7f439c00b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b3fff640 1 -- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c005180 msgr2=0x7f439c00b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c005180 0x7f439c00b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b3fff640 1 -- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f439c0047c0 msgr2=0x7f439c004c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f439c0047c0 0x7f439c004c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.218+0000 7f43b3fff640 1 -- 172.21.15.161:0/3378983160 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f439c003520 con 0x7f439c003b30 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.219+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f439c003b30 0x7f439c003f30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f43b40d0240 tx=0x7f439c0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f42916b19a813064 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.219+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a4004030 con 0x7f439c003b30 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.219+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43a4004190 con 0x7f439c003b30 2024-03-23T14:37:18.218 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.219+0000 7f43b2ffd640 1 -- 172.21.15.161:0/3378983160 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a4004520 con 0x7f439c003b30 2024-03-23T14:37:18.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.219+0000 7f43ba784640 1 -- 172.21.15.161:0/3378983160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f439c003b30 msgr2=0x7f439c003f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.219+0000 7f43ba784640 1 --2- 172.21.15.161:0/3378983160 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f439c003b30 0x7f439c003f30 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f43b40d0240 tx=0x7f439c0057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.219+0000 7f43b8d21640 1 -- 172.21.15.161:0/3378983160 reap_dead start 2024-03-23T14:37:18.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.220+0000 7f43ba784640 1 -- 172.21.15.161:0/3378983160 shutdown_connections 2024-03-23T14:37:18.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.220+0000 7f43ba784640 1 -- 172.21.15.161:0/3378983160 >> 172.21.15.161:0/3378983160 conn(0x7f43b40b1ed0 msgr2=0x7f43b40b22e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:18.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.220+0000 7f43ba784640 1 -- 172.21.15.161:0/3378983160 shutdown_connections 2024-03-23T14:37:18.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.220+0000 7f43ba784640 1 -- 172.21.15.161:0/3378983160 wait complete. 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.221+0000 7f43ba784640 1 Processor -- start 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.221+0000 7f43ba784640 1 -- start start 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43ba784640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c003b30 0x7f43b40c9340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43ba784640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b40b5f20 0x7f43b40b40e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43ba784640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b40c9880 0x7f43b4153630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43ba784640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f43b4159870 con 0x7f43b40c9880 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43ba784640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f43b40c8630 con 0x7f43b40b5f20 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43ba784640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f43b41531a0 con 0x7f439c003b30 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43b8d21640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b40c9880 0x7f43b4153630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.222 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43b8d21640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b40c9880 0x7f43b4153630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37824/0 (socket says 172.21.15.161:37824) 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43b8d21640 1 -- 172.21.15.161:0/3902278573 learned_addr learned my addr 172.21.15.161:0/3902278573 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43b37fe640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b40b5f20 0x7f43b40b40e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.222+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c003b30 0x7f43b40c9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b8d21640 1 -- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c003b30 msgr2=0x7f43b40c9340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b8d21640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c003b30 0x7f43b40c9340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b8d21640 1 -- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b40b5f20 msgr2=0x7f43b40b40e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b8d21640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b40b5f20 0x7f43b40b40e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b8d21640 1 -- 172.21.15.161:0/3902278573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43b40c9da0 con 0x7f43b40c9880 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c003b30 0x7f43b40c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b37fe640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b40b5f20 0x7f43b40b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b8d21640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b40c9880 0x7f43b4153630 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f43b40cb060 tx=0x7f43ac00fd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b0ff9640 1 -- 172.21.15.161:0/3902278573 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac006d60 con 0x7f43b40c9880 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b0ff9640 1 -- 172.21.15.161:0/3902278573 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43ac003970 con 0x7f43b40c9880 2024-03-23T14:37:18.223 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43b40c8880 con 0x7f43b40c9880 2024-03-23T14:37:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43b0ff9640 1 -- 172.21.15.161:0/3902278573 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43ac003ce0 con 0x7f43b40c9880 2024-03-23T14:37:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.223+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43b415a730 con 0x7f43b40c9880 2024-03-23T14:37:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.224+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43b409e530 con 0x7f43b40c9880 2024-03-23T14:37:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.224+0000 7f43b0ff9640 1 -- 172.21.15.161:0/3902278573 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f43ac0050d0 con 0x7f43b40c9880 2024-03-23T14:37:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.225+0000 7f43b0ff9640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f43a006b810 0x7f43a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.225+0000 7f43b0ff9640 1 -- 172.21.15.161:0/3902278573 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f43ac0898c0 con 0x7f43b40c9880 2024-03-23T14:37:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.225+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f43a006b810 0x7f43a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.226+0000 7f43b3fff640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f43a006b810 0x7f43a006dcd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f43a4035000 tx=0x7f43a4030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.229+0000 7f43b0ff9640 1 -- 172.21.15.161:0/3902278573 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f43ac054370 con 0x7f43b40c9880 2024-03-23T14:37:18.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.404+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f43b4071fe0 con 0x7f43b40c9880 2024-03-23T14:37:18.400 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.404+0000 7f43b0ff9640 1 -- 172.21.15.161:0/3902278573 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f43ac0190a0 con 0x7f43b40c9880 2024-03-23T14:37:18.401 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f43a006b810 msgr2=0x7f43a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f43a006b810 0x7f43a006dcd0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f43a4035000 tx=0x7f43a4030000 comp rx=0 tx=0).stop 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b40c9880 msgr2=0x7f43b4153630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b40c9880 0x7f43b4153630 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f43b40cb060 tx=0x7f43ac00fd60 comp rx=0 tx=0).stop 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 shutdown_connections 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f43a006b810 0x7f43a006dcd0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f43b40c9880 0x7f43b4153630 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f43b40b5f20 0x7f43b40b40e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 --2- 172.21.15.161:0/3902278573 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f439c003b30 0x7f43b40c9340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.406+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 >> 172.21.15.161:0/3902278573 conn(0x7f43b40b1ed0 msgr2=0x7f43b40af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.407+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 shutdown_connections 2024-03-23T14:37:18.403 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.407+0000 7f43ba784640 1 -- 172.21.15.161:0/3902278573 wait complete. 2024-03-23T14:37:18.413 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:37:18.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.523+0000 7fc5e7409640 1 Processor -- start 2024-03-23T14:37:18.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.523+0000 7fc5e7409640 1 -- start start 2024-03-23T14:37:18.520 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e7409640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc5e0161820 0x7fc5e00d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e7409640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d1270 0x7fc5e00d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e7409640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc5e00d29d0 0x7fc5e0161e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e7409640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fc5e0161c00 con 0x7fc5e00d29d0 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e7409640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc5e00d2e20 con 0x7fc5e0161820 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e7409640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e00cea30 con 0x7fc5e00d1270 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e59a6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc5e00d29d0 0x7fc5e0161e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e59a6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fc5e00d29d0 0x7fc5e0161e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37832/0 (socket says 172.21.15.161:37832) 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e59a6640 1 -- 172.21.15.161:0/1967937897 learned_addr learned my addr 172.21.15.161:0/1967937897 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e51a5640 1 --2- 172.21.15.161:0/1967937897 >> v2:172.21.15.183:3300/0 conn(0x7fc5e0161820 0x7fc5e00d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e49a4640 1 --2- 172.21.15.161:0/1967937897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d1270 0x7fc5e00d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e59a6640 1 -- 172.21.15.161:0/1967937897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d1270 msgr2=0x7fc5e00d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:18.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e59a6640 1 --2- 172.21.15.161:0/1967937897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d1270 0x7fc5e00d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e59a6640 1 -- 172.21.15.161:0/1967937897 >> v2:172.21.15.183:3300/0 conn(0x7fc5e0161820 msgr2=0x7fc5e00d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:18.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e59a6640 1 --2- 172.21.15.161:0/1967937897 >> v2:172.21.15.183:3300/0 conn(0x7fc5e0161820 0x7fc5e00d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e59a6640 1 -- 172.21.15.161:0/1967937897 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5e00cfd70 con 0x7fc5e00d29d0 2024-03-23T14:37:18.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e49a4640 1 --2- 172.21.15.161:0/1967937897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d1270 0x7fc5e00d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:18.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.524+0000 7fc5e51a5640 1 --2- 172.21.15.161:0/1967937897 >> v2:172.21.15.183:3300/0 conn(0x7fc5e0161820 0x7fc5e00d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:18.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.525+0000 7fc5e59a6640 1 --2- 172.21.15.161:0/1967937897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d29d0 0x7fc5e0161e00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc5e006b270 tx=0x7fc5d802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ecb4d7b2f6aab7ae server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5d7fff640 1 -- 172.21.15.161:0/1967937897 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d802ca70 con 0x7fc5e00d29d0 2024-03-23T14:37:18.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5d7fff640 1 -- 172.21.15.161:0/1967937897 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5d802cbd0 con 0x7fc5e00d29d0 2024-03-23T14:37:18.526 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5d7fff640 1 -- 172.21.15.161:0/1967937897 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5d80343f0 con 0x7fc5e00d29d0 2024-03-23T14:37:18.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 -- 172.21.15.161:0/1967937897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d29d0 msgr2=0x7fc5e0161e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 --2- 172.21.15.161:0/1967937897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d29d0 0x7fc5e0161e00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7fc5e006b270 tx=0x7fc5d802bed0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 -- 172.21.15.161:0/1967937897 shutdown_connections 2024-03-23T14:37:18.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 --2- 172.21.15.161:0/1967937897 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d29d0 0x7fc5e0161e00 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 --2- 172.21.15.161:0/1967937897 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d1270 0x7fc5e00d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 --2- 172.21.15.161:0/1967937897 >> v2:172.21.15.183:3300/0 conn(0x7fc5e0161820 0x7fc5e00d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 -- 172.21.15.161:0/1967937897 >> 172.21.15.161:0/1967937897 conn(0x7fc5e009d190 msgr2=0x7fc5e00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:18.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 -- 172.21.15.161:0/1967937897 shutdown_connections 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.526+0000 7fc5e7409640 1 -- 172.21.15.161:0/1967937897 wait complete. 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.527+0000 7fc5e7409640 1 Processor -- start 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.527+0000 7fc5e7409640 1 -- start start 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e7409640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d1270 0x7fc5e00b93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e7409640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d29d0 0x7fc5e00ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e7409640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5e0161820 0x7fc5e014d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e7409640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e00c79e0 con 0x7fc5e00d1270 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e7409640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc5e00b98e0 con 0x7fc5e0161820 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e7409640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc5e00c7bf0 con 0x7fc5e00d29d0 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e51a5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d1270 0x7fc5e00b93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.528 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e51a5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d1270 0x7fc5e00b93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37836/0 (socket says 172.21.15.161:37836) 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e51a5640 1 -- 172.21.15.161:0/3029761356 learned_addr learned my addr 172.21.15.161:0/3029761356 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.528+0000 7fc5e59a6640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5e0161820 0x7fc5e014d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e49a4640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d29d0 0x7fc5e00ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e51a5640 1 -- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d29d0 msgr2=0x7fc5e00ca210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e51a5640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d29d0 0x7fc5e00ca210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e51a5640 1 -- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5e0161820 msgr2=0x7fc5e014d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e51a5640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5e0161820 0x7fc5e014d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e51a5640 1 -- 172.21.15.161:0/3029761356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5d8009b30 con 0x7fc5e00d1270 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e59a6640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5e0161820 0x7fc5e014d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:18.529 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e49a4640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d29d0 0x7fc5e00ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:18.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e51a5640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d1270 0x7fc5e00b93a0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00d800 tx=0x7fc5dc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5d5ffb640 1 -- 172.21.15.161:0/3029761356 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc004030 con 0x7fc5e00d1270 2024-03-23T14:37:18.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5d5ffb640 1 -- 172.21.15.161:0/3029761356 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5dc004190 con 0x7fc5e00d1270 2024-03-23T14:37:18.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5e00c7ed0 con 0x7fc5e00d1270 2024-03-23T14:37:18.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5d5ffb640 1 -- 172.21.15.161:0/3029761356 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5dc004520 con 0x7fc5e00d1270 2024-03-23T14:37:18.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.529+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5e015b0e0 con 0x7fc5e00d1270 2024-03-23T14:37:18.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.530+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5e00d2270 con 0x7fc5e00d1270 2024-03-23T14:37:18.530 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.531+0000 7fc5d5ffb640 1 -- 172.21.15.161:0/3029761356 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc5dc0096a0 con 0x7fc5e00d1270 2024-03-23T14:37:18.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.531+0000 7fc5d5ffb640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc5c406b810 0x7fc5c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.531+0000 7fc5d5ffb640 1 -- 172.21.15.161:0/3029761356 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fc5dc08a390 con 0x7fc5e00d1270 2024-03-23T14:37:18.531 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.531+0000 7fc5e49a4640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc5c406b810 0x7fc5c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.532+0000 7fc5e49a4640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc5c406b810 0x7fc5c406dcd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc5e00ce3d0 tx=0x7fc5d0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.532 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.535+0000 7fc5d5ffb640 1 -- 172.21.15.161:0/3029761356 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc5dc054db0 con 0x7fc5e00d1270 2024-03-23T14:37:18.719 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.723+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc5e00d45c0 con 0x7fc5e00d1270 2024-03-23T14:37:18.720 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.724+0000 7fc5d5ffb640 1 -- 172.21.15.161:0/3029761356 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7fc5dc058470 con 0x7fc5e00d1270 2024-03-23T14:37:18.720 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc5c406b810 msgr2=0x7fc5c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc5c406b810 0x7fc5c406dcd0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fc5e00ce3d0 tx=0x7fc5d0008040 comp rx=0 tx=0).stop 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d1270 msgr2=0x7fc5e00b93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d1270 0x7fc5e00b93a0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7fc5dc00d800 tx=0x7fc5dc00dcc0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 shutdown_connections 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc5c406b810 0x7fc5c406dcd0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc5e0161820 0x7fc5e014d040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc5e00d29d0 0x7fc5e00ca210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.722 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 --2- 172.21.15.161:0/3029761356 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc5e00d1270 0x7fc5e00b93a0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 >> 172.21.15.161:0/3029761356 conn(0x7fc5e009d190 msgr2=0x7fc5e00b0b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:18.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 shutdown_connections 2024-03-23T14:37:18.723 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.726+0000 7fc5e7409640 1 -- 172.21.15.161:0/3029761356 wait complete. 2024-03-23T14:37:18.732 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:37:18.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.841+0000 7f05b92f1640 1 Processor -- start 2024-03-23T14:37:18.837 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.842+0000 7f05b92f1640 1 -- start start 2024-03-23T14:37:18.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.842+0000 7f05b92f1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 0x7f05b415b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.839 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.842+0000 7f05b92f1640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05b40724e0 0x7f05b406e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.842+0000 7f05b92f1640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f05b415b930 0x7f05b406ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.842+0000 7f05b92f1640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f05b415c430 con 0x7f05b40724e0 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.842+0000 7f05b92f1640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f05b415be70 con 0x7f05b415b930 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.842+0000 7f05b92f1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f05b415c150 con 0x7f05b4072100 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b27fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05b40724e0 0x7f05b406e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b27fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f05b40724e0 0x7f05b406e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37840/0 (socket says 172.21.15.161:37840) 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b27fc640 1 -- 172.21.15.161:0/2799570039 learned_addr learned my addr 172.21.15.161:0/2799570039 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b2ffd640 1 --2- 172.21.15.161:0/2799570039 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 0x7f05b415b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b37fe640 1 --2- 172.21.15.161:0/2799570039 >> v2:172.21.15.183:3300/0 conn(0x7f05b415b930 0x7f05b406ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b27fc640 1 -- 172.21.15.161:0/2799570039 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 msgr2=0x7f05b415b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b27fc640 1 --2- 172.21.15.161:0/2799570039 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 0x7f05b415b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b27fc640 1 -- 172.21.15.161:0/2799570039 >> v2:172.21.15.183:3300/0 conn(0x7f05b415b930 msgr2=0x7f05b406ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b27fc640 1 --2- 172.21.15.161:0/2799570039 >> v2:172.21.15.183:3300/0 conn(0x7f05b415b930 0x7f05b406ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.840 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.843+0000 7f05b27fc640 1 -- 172.21.15.161:0/2799570039 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05b40b2c70 con 0x7f05b40724e0 2024-03-23T14:37:18.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b27fc640 1 --2- 172.21.15.161:0/2799570039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b406e590 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f05b40cce40 tx=0x7f05a802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3aee82d11617d38 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b1ffb640 1 -- 172.21.15.161:0/2799570039 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8031870 con 0x7f05b40724e0 2024-03-23T14:37:18.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b1ffb640 1 -- 172.21.15.161:0/2799570039 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a80319d0 con 0x7f05b40724e0 2024-03-23T14:37:18.841 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b1ffb640 1 -- 172.21.15.161:0/2799570039 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8031d60 con 0x7f05b40724e0 2024-03-23T14:37:18.843 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 -- 172.21.15.161:0/2799570039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 msgr2=0x7f05b406e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 --2- 172.21.15.161:0/2799570039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b406e590 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f05b40cce40 tx=0x7f05a802b910 comp rx=0 tx=0).stop 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 -- 172.21.15.161:0/2799570039 shutdown_connections 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 --2- 172.21.15.161:0/2799570039 >> v2:172.21.15.183:3300/0 conn(0x7f05b415b930 0x7f05b406ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 --2- 172.21.15.161:0/2799570039 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b406e590 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 --2- 172.21.15.161:0/2799570039 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 0x7f05b415b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 -- 172.21.15.161:0/2799570039 >> 172.21.15.161:0/2799570039 conn(0x7f05b40b17f0 msgr2=0x7f05b40b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 -- 172.21.15.161:0/2799570039 shutdown_connections 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.844+0000 7f05b92f1640 1 -- 172.21.15.161:0/2799570039 wait complete. 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.846+0000 7f05b92f1640 1 Processor -- start 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.846+0000 7f05b92f1640 1 -- start start 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.846+0000 7f05b92f1640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 0x7f05b41453c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.846+0000 7f05b92f1640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b4157c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.846+0000 7f05b92f1640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b415b930 0x7f05b4158140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.844 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.846+0000 7f05b92f1640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f05b4156450 con 0x7f05b40724e0 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.846+0000 7f05b92f1640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f05b4156690 con 0x7f05b415b930 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.846+0000 7f05b92f1640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f05b4156970 con 0x7f05b4072100 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b4157c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b4157c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37842/0 (socket says 172.21.15.161:37842) 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 -- 172.21.15.161:0/3753119017 learned_addr learned my addr 172.21.15.161:0/3753119017 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b2ffd640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 0x7f05b41453c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b37fe640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b415b930 0x7f05b4158140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 -- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 msgr2=0x7f05b41453c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 0x7f05b41453c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 -- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b415b930 msgr2=0x7f05b4158140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b415b930 0x7f05b4158140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 -- 172.21.15.161:0/3753119017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a80094d0 con 0x7f05b40724e0 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b27fc640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b4157c00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f05a802b950 tx=0x7f05a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.845 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f059b7fe640 1 -- 172.21.15.161:0/3753119017 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a803ac60 con 0x7f05b40724e0 2024-03-23T14:37:18.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f059b7fe640 1 -- 172.21.15.161:0/3753119017 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f05a803adc0 con 0x7f05b40724e0 2024-03-23T14:37:18.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f059b7fe640 1 -- 172.21.15.161:0/3753119017 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05a8044630 con 0x7f05b40724e0 2024-03-23T14:37:18.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05b4156c50 con 0x7f05b40724e0 2024-03-23T14:37:18.846 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.847+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05b4145900 con 0x7f05b40724e0 2024-03-23T14:37:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.848+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05b4157170 con 0x7f05b40724e0 2024-03-23T14:37:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.849+0000 7f059b7fe640 1 -- 172.21.15.161:0/3753119017 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f05a804d450 con 0x7f05b40724e0 2024-03-23T14:37:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.849+0000 7f059b7fe640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f05a006b810 0x7f05a006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.849+0000 7f059b7fe640 1 -- 172.21.15.161:0/3753119017 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f05a802d040 con 0x7f05b40724e0 2024-03-23T14:37:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.849+0000 7f05b2ffd640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f05a006b810 0x7f05a006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:18.847 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.850+0000 7f05b2ffd640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f05a006b810 0x7f05a006dcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009a50 tx=0x7f05a4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:18.849 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:18.853+0000 7f059b7fe640 1 -- 172.21.15.161:0/3753119017 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f05b4157170 con 0x7f05b40724e0 2024-03-23T14:37:19.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.072+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f05b4157170 con 0x7f05b40724e0 2024-03-23T14:37:19.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.073+0000 7f059b7fe640 1 -- 172.21.15.161:0/3753119017 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) ==== 80+0+0 (secure 0 0 0) 0x7f05b4157170 con 0x7f05b40724e0 2024-03-23T14:37:19.068 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f05a006b810 msgr2=0x7f05a006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f05a006b810 0x7f05a006dcd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f05a4009a50 tx=0x7f05a4008040 comp rx=0 tx=0).stop 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 msgr2=0x7f05b4157c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b4157c00 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f05a802b950 tx=0x7f05a80046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 shutdown_connections 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f05a006b810 0x7f05a006dcd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f05b415b930 0x7f05b4158140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f05b40724e0 0x7f05b4157c00 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 --2- 172.21.15.161:0/3753119017 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f05b4072100 0x7f05b41453c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 >> 172.21.15.161:0/3753119017 conn(0x7f05b40b17f0 msgr2=0x7f05b40af760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 shutdown_connections 2024-03-23T14:37:19.071 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.075+0000 7f05b92f1640 1 -- 172.21.15.161:0/3753119017 wait complete. 2024-03-23T14:37:19.085 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:37:19.085 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:19.101 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:19.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.262+0000 7fd285fe9640 1 Processor -- start 2024-03-23T14:37:19.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.262+0000 7fd285fe9640 1 -- start start 2024-03-23T14:37:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.262+0000 7fd285fe9640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fd2800d1470 0x7fd2800d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd285fe9640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2800d0460 con 0x7fd28015e400 2024-03-23T14:37:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd285fe9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd2800d06b0 con 0x7fd2800d2970 2024-03-23T14:37:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd285fe9640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fd2800d1050 con 0x7fd2800d1470 2024-03-23T14:37:19.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27f7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd28015e400 0x7fd2800cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44514/0 (socket says 172.21.15.161:44514) 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27f7fe640 1 -- 172.21.15.161:0/2873938162 learned_addr learned my addr 172.21.15.161:0/2873938162 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27effd640 1 --2- 172.21.15.161:0/2873938162 >> v2:172.21.15.183:3300/0 conn(0x7fd2800d1470 0x7fd2800d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 469350002 0 0) 0x7fd2800d0460 con 0x7fd28015e400 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd268003520 con 0x7fd28015e400 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1093525528 0 0) 0x7fd2800d06b0 con 0x7fd2800d2970 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd2800d0460 con 0x7fd2800d2970 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4097415110 0 0) 0x7fd268003520 con 0x7fd28015e400 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 >> v2:172.21.15.183:3300/0 conn(0x7fd2800d1470 msgr2=0x7fd2800d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 --2- 172.21.15.161:0/2873938162 >> v2:172.21.15.183:3300/0 conn(0x7fd2800d1470 0x7fd2800d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 >> v1:172.21.15.183:6789/0 conn(0x7fd2800d2970 legacy=0x7fd2800b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2800d0990 con 0x7fd28015e400 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fd270001670 con 0x7fd28015e400 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27effd640 1 --2- 172.21.15.161:0/2873938162 >> v2:172.21.15.183:3300/0 conn(0x7fd2800d1470 0x7fd2800d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:19.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 >> v1:172.21.15.161:6789/0 conn(0x7fd28015e400 legacy=0x7fd2800cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd268003b00 0x7fd268003f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2680047c0 0x7fd268004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd268005180 0x7fd26800b980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2800d1050 con 0x7fd268005180 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd26800c000 con 0x7fd2680047c0 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd26800c1c0 con 0x7fd268003b00 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.263+0000 7fd27ffff640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd268005180 0x7fd26800b980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27effd640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2680047c0 0x7fd268004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27f7fe640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd268003b00 0x7fd268003f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27effd640 1 -- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd268003b00 msgr2=0x7fd268003f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27effd640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd268003b00 0x7fd268003f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27effd640 1 -- 172.21.15.161:0/2873938162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd268005180 msgr2=0x7fd26800b980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27effd640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd268005180 0x7fd26800b980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27effd640 1 -- 172.21.15.161:0/2873938162 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd268003520 con 0x7fd2680047c0 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27ffff640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd268005180 0x7fd26800b980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.264+0000 7fd27ffff640 1 -- 172.21.15.161:0/2873938162 reap_dead start 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd27effd640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2680047c0 0x7fd268004c40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd274009c90 tx=0x7fd2680057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eae5f776141ead31 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27402c8f0 con 0x7fd2680047c0 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd27402ca50 con 0x7fd2680047c0 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd27e7fc640 1 -- 172.21.15.161:0/2873938162 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd27402ce40 con 0x7fd2680047c0 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd285fe9640 1 -- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2680047c0 msgr2=0x7fd268004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd285fe9640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2680047c0 0x7fd268004c40 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd274009c90 tx=0x7fd2680057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd285fe9640 1 -- 172.21.15.161:0/2873938162 shutdown_connections 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd285fe9640 1 --2- 172.21.15.161:0/2873938162 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd2680047c0 0x7fd268004c40 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd285fe9640 1 -- 172.21.15.161:0/2873938162 >> 172.21.15.161:0/2873938162 conn(0x7fd28009d190 msgr2=0x7fd2800d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd285fe9640 1 -- 172.21.15.161:0/2873938162 shutdown_connections 2024-03-23T14:37:19.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.265+0000 7fd285fe9640 1 -- 172.21.15.161:0/2873938162 wait complete. 2024-03-23T14:37:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.266+0000 7fd285fe9640 1 Processor -- start 2024-03-23T14:37:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.267+0000 7fd285fe9640 1 -- start start 2024-03-23T14:37:19.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.267+0000 7fd285fe9640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2680047c0 0x7fd28015b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.267+0000 7fd285fe9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015f210 0x7fd2800ca570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.267+0000 7fd285fe9640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280160c50 0x7fd2801610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.267+0000 7fd285fe9640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd2800b81e0 con 0x7fd28015f210 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.267+0000 7fd285fe9640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd2800b7610 con 0x7fd280160c50 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.267+0000 7fd285fe9640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd28015b5e0 con 0x7fd2680047c0 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.267+0000 7fd27effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015f210 0x7fd2800ca570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015f210 0x7fd2800ca570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37860/0 (socket says 172.21.15.161:37860) 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27effd640 1 -- 172.21.15.161:0/3864069833 learned_addr learned my addr 172.21.15.161:0/3864069833 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27ffff640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280160c50 0x7fd2801610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27f7fe640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2680047c0 0x7fd28015b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27effd640 1 -- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2680047c0 msgr2=0x7fd28015b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27effd640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2680047c0 0x7fd28015b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27effd640 1 -- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280160c50 msgr2=0x7fd2801610d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27effd640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280160c50 0x7fd2801610d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27effd640 1 -- 172.21.15.161:0/3864069833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2800b78b0 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd27effd640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015f210 0x7fd2800ca570 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd274009c90 tx=0x7fd274004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd267fff640 1 -- 172.21.15.161:0/3864069833 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274034c80 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd267fff640 1 -- 172.21.15.161:0/3864069833 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd274034de0 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2800b7b90 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd267fff640 1 -- 172.21.15.161:0/3864069833 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd274004240 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.268+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd280161610 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.269+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2800b8350 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.270+0000 7fd267fff640 1 -- 172.21.15.161:0/3864069833 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd27403b050 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.270+0000 7fd267fff640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26c06b810 0x7fd26c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.270+0000 7fd267fff640 1 -- 172.21.15.161:0/3864069833 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fd2740abbc0 con 0x7fd28015f210 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.270+0000 7fd27f7fe640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26c06b810 0x7fd26c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.271+0000 7fd27f7fe640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26c06b810 0x7fd26c06dcd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd270001eb0 tx=0x7fd270001e00 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.274+0000 7fd267fff640 1 -- 172.21.15.161:0/3864069833 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd274076750 con 0x7fd28015f210 2024-03-23T14:37:19.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.453+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2800a1e00 con 0x7fd28015f210 2024-03-23T14:37:19.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.454+0000 7fd267fff640 1 -- 172.21.15.161:0/3864069833 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7fd27404d020 con 0x7fd28015f210 2024-03-23T14:37:19.450 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:19.450 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:19.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26c06b810 msgr2=0x7fd26c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26c06b810 0x7fd26c06dcd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fd270001eb0 tx=0x7fd270001e00 comp rx=0 tx=0).stop 2024-03-23T14:37:19.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015f210 msgr2=0x7fd2800ca570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015f210 0x7fd2800ca570 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7fd274009c90 tx=0x7fd274004730 comp rx=0 tx=0).stop 2024-03-23T14:37:19.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 shutdown_connections 2024-03-23T14:37:19.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd26c06b810 0x7fd26c06dcd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd280160c50 0x7fd2801610d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd28015f210 0x7fd2800ca570 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 --2- 172.21.15.161:0/3864069833 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd2680047c0 0x7fd28015b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 >> 172.21.15.161:0/3864069833 conn(0x7fd28009d190 msgr2=0x7fd2800b16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:19.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 shutdown_connections 2024-03-23T14:37:19.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.456+0000 7fd285fe9640 1 -- 172.21.15.161:0/3864069833 wait complete. 2024-03-23T14:37:19.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:56.747667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:57.478793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:37:19.464 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:37:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.572+0000 7fa5fd455640 1 Processor -- start 2024-03-23T14:37:19.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.572+0000 7fa5fd455640 1 -- start start 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5fd455640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80727d0 0x7fa5f815b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5fd455640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5f80cbf10 con 0x7fa5f8071b80 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5fd455640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5f80cb300 con 0x7fa5f80ce6e0 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5fd455640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f80cb5e0 con 0x7fa5f80727d0 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5f6ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa5f8071b80 0x7fa5f8071f60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44524/0 (socket says 172.21.15.161:44524) 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5f6ffd640 1 -- 172.21.15.161:0/829149060 learned_addr learned my addr 172.21.15.161:0/829149060 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5f67fc640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80727d0 0x7fa5f815b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3031153326 0 0) 0x7fa5f80cbf10 con 0x7fa5f8071b80 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.573+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5e0003520 con 0x7fa5f8071b80 2024-03-23T14:37:19.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3919141588 0 0) 0x7fa5e0003520 con 0x7fa5f8071b80 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80727d0 msgr2=0x7fa5f815b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80727d0 0x7fa5f815b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 >> v1:172.21.15.183:6789/0 conn(0x7fa5f80ce6e0 legacy=0x7fa5f80ceb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f80cb8c0 con 0x7fa5f8071b80 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fa5f00019f0 con 0x7fa5f8071b80 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 >> v1:172.21.15.161:6789/0 conn(0x7fa5f8071b80 legacy=0x7fa5f8071f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e00039a0 0x7fa5e0003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80727d0 0x7fa5e00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e0004d00 0x7fa5e0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f80cb5e0 con 0x7fa5e00039a0 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa5e000bba0 con 0x7fa5e0004d00 2024-03-23T14:37:19.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa5e000bd30 con 0x7fa5f80727d0 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f6ffd640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e00039a0 0x7fa5e0003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f67fc640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80727d0 0x7fa5e00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f77fe640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e0004d00 0x7fa5e0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f6ffd640 1 -- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80727d0 msgr2=0x7fa5e00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f6ffd640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80727d0 0x7fa5e00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f6ffd640 1 -- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e0004d00 msgr2=0x7fa5e0005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f6ffd640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e0004d00 0x7fa5e0005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f6ffd640 1 -- 172.21.15.161:0/829149060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5e0003520 con 0x7fa5e00039a0 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.574+0000 7fa5f77fe640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e0004d00 0x7fa5e0005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.575+0000 7fa5f6ffd640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e00039a0 0x7fa5e0003dc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000a860 tx=0x7fa5e00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41796e8590a98cd9 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.575+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f00072d0 con 0x7fa5e00039a0 2024-03-23T14:37:19.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.575+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5f002f070 con 0x7fa5e00039a0 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.575+0000 7fa5f5ffb640 1 -- 172.21.15.161:0/829149060 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5f00076b0 con 0x7fa5e00039a0 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.576+0000 7fa5fd455640 1 -- 172.21.15.161:0/829149060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e00039a0 msgr2=0x7fa5e0003dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.576+0000 7fa5fd455640 1 --2- 172.21.15.161:0/829149060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5e00039a0 0x7fa5e0003dc0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7fa5f000a860 tx=0x7fa5e00057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.576+0000 7fa5f77fe640 1 -- 172.21.15.161:0/829149060 reap_dead start 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.576+0000 7fa5fd455640 1 -- 172.21.15.161:0/829149060 shutdown_connections 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.576+0000 7fa5fd455640 1 -- 172.21.15.161:0/829149060 >> 172.21.15.161:0/829149060 conn(0x7fa5f80d0b10 msgr2=0x7fa5f80d0f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.576+0000 7fa5fd455640 1 -- 172.21.15.161:0/829149060 shutdown_connections 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.576+0000 7fa5fd455640 1 -- 172.21.15.161:0/829149060 wait complete. 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.577+0000 7fa5fd455640 1 Processor -- start 2024-03-23T14:37:19.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.577+0000 7fa5fd455640 1 -- start start 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5fd455640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e00039a0 0x7fa5f80b5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5fd455640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f8158f10 0x7fa5f80af270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5fd455640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80b55b0 0x7fa5f815c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5fd455640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f80b6010 con 0x7fa5f8158f10 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5fd455640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa5f80c8ac0 con 0x7fa5e00039a0 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5fd455640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa5f80b5a50 con 0x7fa5f80b55b0 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f8158f10 0x7fa5f80af270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f8158f10 0x7fa5f80af270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37874/0 (socket says 172.21.15.161:37874) 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 -- 172.21.15.161:0/3926804928 learned_addr learned my addr 172.21.15.161:0/3926804928 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 -- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80b55b0 msgr2=0x7fa5f815c3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f77fe640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80b55b0 0x7fa5f815c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80b55b0 0x7fa5f815c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 -- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e00039a0 msgr2=0x7fa5f80b5070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f6ffd640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e00039a0 0x7fa5f80b5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e00039a0 0x7fa5f80b5070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 -- 172.21.15.161:0/3926804928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5f80b5d30 con 0x7fa5f8158f10 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5f67fc640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f8158f10 0x7fa5f80af270 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00b7f0 tx=0x7fa5ec00bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.578+0000 7fa5df7fe640 1 -- 172.21.15.161:0/3926804928 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec010830 con 0x7fa5f8158f10 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.579+0000 7fa5df7fe640 1 -- 172.21.15.161:0/3926804928 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa5ec010990 con 0x7fa5f8158f10 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.579+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5f80c8750 con 0x7fa5f8158f10 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.579+0000 7fa5df7fe640 1 -- 172.21.15.161:0/3926804928 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5ec010d60 con 0x7fa5f8158f10 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.579+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5f80af990 con 0x7fa5f8158f10 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.580+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5f804f4e0 con 0x7fa5f8158f10 2024-03-23T14:37:19.580 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.580+0000 7fa5df7fe640 1 -- 172.21.15.161:0/3926804928 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa5ec01d450 con 0x7fa5f8158f10 2024-03-23T14:37:19.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.580+0000 7fa5df7fe640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5e406b810 0x7fa5e406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.581+0000 7fa5df7fe640 1 -- 172.21.15.161:0/3926804928 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fa5ec08aa20 con 0x7fa5f8158f10 2024-03-23T14:37:19.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.581+0000 7fa5f6ffd640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5e406b810 0x7fa5e406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.581+0000 7fa5f6ffd640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5e406b810 0x7fa5e406dcd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0034000 tx=0x7fa5f0007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.581 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.585+0000 7fa5df7fe640 1 -- 172.21.15.161:0/3926804928 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fa5ec055440 con 0x7fa5f8158f10 2024-03-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.761+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa5f80c8c00 con 0x7fa5f8158f10 2024-03-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.762+0000 7fa5df7fe640 1 -- 172.21.15.161:0/3926804928 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7fa5ec0220a0 con 0x7fa5f8158f10 2024-03-23T14:37:19.757 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5e406b810 msgr2=0x7fa5e406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5e406b810 0x7fa5e406dcd0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0034000 tx=0x7fa5f0007000 comp rx=0 tx=0).stop 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f8158f10 msgr2=0x7fa5f80af270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f8158f10 0x7fa5f80af270 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec00b7f0 tx=0x7fa5ec00bcb0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 shutdown_connections 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa5e406b810 0x7fa5e406dcd0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa5f80b55b0 0x7fa5f815c3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa5f8158f10 0x7fa5f80af270 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 --2- 172.21.15.161:0/3926804928 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa5e00039a0 0x7fa5f80b5070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.760 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 >> 172.21.15.161:0/3926804928 conn(0x7fa5f80d0b10 msgr2=0x7fa5f80b1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:19.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 shutdown_connections 2024-03-23T14:37:19.761 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.764+0000 7fa5fd455640 1 -- 172.21.15.161:0/3926804928 wait complete. 2024-03-23T14:37:19.771 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:37:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.879+0000 7f112d981640 1 Processor -- start 2024-03-23T14:37:19.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.879+0000 7f112d981640 1 -- start start 2024-03-23T14:37:19.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.880+0000 7f112d981640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 0x7f112806e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.880+0000 7f112d981640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1128070c20 0x7f112815b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.880+0000 7f112d981640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11280b2c20 con 0x7f11280710a0 2024-03-23T14:37:19.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.880+0000 7f112d981640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f112806ee90 con 0x7f1128070c20 2024-03-23T14:37:19.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.880+0000 7f112d981640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f11280b2a10 con 0x7f11281549c0 2024-03-23T14:37:19.883 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.880+0000 7f11277fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f11280710a0 0x7f112815e030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44526/0 (socket says 172.21.15.161:44526) 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.880+0000 7f11277fe640 1 -- 172.21.15.161:0/2033456590 learned_addr learned my addr 172.21.15.161:0/2033456590 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4280865793 0 0) 0x7f11280b2c20 con 0x7f11280710a0 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f11267fc640 1 --2- 172.21.15.161:0/2033456590 >> v2:172.21.15.183:3300/0 conn(0x7f1128070c20 0x7f112815b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1114004630 con 0x7f11280710a0 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1126ffd640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 0x7f112806e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 795766489 0 0) 0x7f1114004630 con 0x7f11280710a0 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 msgr2=0x7f112806e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 0x7f112806e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 >> v2:172.21.15.183:3300/0 conn(0x7f1128070c20 msgr2=0x7f112815b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 --2- 172.21.15.161:0/2033456590 >> v2:172.21.15.183:3300/0 conn(0x7f1128070c20 0x7f112815b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11280b2f00 con 0x7f11280710a0 2024-03-23T14:37:19.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f11200019f0 con 0x7f11280710a0 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1126ffd640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 0x7f112806e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 >> v1:172.21.15.161:6789/0 conn(0x7f11280710a0 legacy=0x7f112815e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1114004ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140058d0 0x7f1114005d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 0x7f1114006290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f11280b2a10 con 0x7f1114004a80 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f111400ca90 con 0x7f11140058d0 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1114003440 con 0x7f11281549c0 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1126ffd640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1114004ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f11267fc640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140058d0 0x7f1114005d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f11277fe640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 0x7f1114006290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1126ffd640 1 -- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 msgr2=0x7f1114006290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1126ffd640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f11281549c0 0x7f1114006290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1126ffd640 1 -- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140058d0 msgr2=0x7f1114005d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1126ffd640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f11140058d0 0x7f1114005d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.881+0000 7f1126ffd640 1 -- 172.21.15.161:0/2033456590 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1114004630 con 0x7f1114004a80 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.882+0000 7f1126ffd640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1114004ea0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f11100098a0 tx=0x7f1114006900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7975d140665f3e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.882+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110004230 con 0x7f1114004a80 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.882+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1110004390 con 0x7f1114004a80 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.882+0000 7f1125ffb640 1 -- 172.21.15.161:0/2033456590 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110003830 con 0x7f1114004a80 2024-03-23T14:37:19.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.883+0000 7f112d981640 1 -- 172.21.15.161:0/2033456590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 msgr2=0x7f1114004ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.883+0000 7f112d981640 1 --2- 172.21.15.161:0/2033456590 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1114004ea0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f11100098a0 tx=0x7f1114006900 comp rx=0 tx=0).stop 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.883+0000 7f11277fe640 1 -- 172.21.15.161:0/2033456590 reap_dead start 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.883+0000 7f112d981640 1 -- 172.21.15.161:0/2033456590 shutdown_connections 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.883+0000 7f112d981640 1 -- 172.21.15.161:0/2033456590 >> 172.21.15.161:0/2033456590 conn(0x7f112806b4a0 msgr2=0x7f112806b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.883+0000 7f112d981640 1 -- 172.21.15.161:0/2033456590 shutdown_connections 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.883+0000 7f112d981640 1 -- 172.21.15.161:0/2033456590 wait complete. 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.884+0000 7f112d981640 1 Processor -- start 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.884+0000 7f112d981640 1 -- start start 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f112d981640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1128152b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f112d981640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1128150b80 0x7f11280b0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f112d981640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f112813f040 0x7f112813f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f112d981640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f11281f0890 con 0x7f1114004a80 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f112d981640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1128153040 con 0x7f1128150b80 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f112d981640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f112813f9e0 con 0x7f112813f040 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f1126ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1128152b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f1126ffd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1128152b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37894/0 (socket says 172.21.15.161:37894) 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f1126ffd640 1 -- 172.21.15.161:0/1170174662 learned_addr learned my addr 172.21.15.161:0/1170174662 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f11267fc640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1128150b80 0x7f11280b0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.885+0000 7f11277fe640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f112813f040 0x7f112813f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f1126ffd640 1 -- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f112813f040 msgr2=0x7f112813f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f1126ffd640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f112813f040 0x7f112813f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f1126ffd640 1 -- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1128150b80 msgr2=0x7f11280b0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f1126ffd640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1128150b80 0x7f11280b0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f1126ffd640 1 -- 172.21.15.161:0/1170174662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11281efd10 con 0x7f1114004a80 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f11277fe640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f112813f040 0x7f112813f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f1126ffd640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1128152b00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f111002bf90 tx=0x7f1110004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f111ffff640 1 -- 172.21.15.161:0/1170174662 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110030630 con 0x7f1114004a80 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f111ffff640 1 -- 172.21.15.161:0/1170174662 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1110030790 con 0x7f1114004a80 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11281efff0 con 0x7f1114004a80 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f111ffff640 1 -- 172.21.15.161:0/1170174662 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1110030b40 con 0x7f1114004a80 2024-03-23T14:37:19.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.886+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11281f05b0 con 0x7f1114004a80 2024-03-23T14:37:19.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.887+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11281f07d0 con 0x7f1114004a80 2024-03-23T14:37:19.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.888+0000 7f111ffff640 1 -- 172.21.15.161:0/1170174662 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1110030ca0 con 0x7f1114004a80 2024-03-23T14:37:19.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.888+0000 7f111ffff640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f110806b810 0x7f110806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:19.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.888+0000 7f111ffff640 1 -- 172.21.15.161:0/1170174662 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f11100abeb0 con 0x7f1114004a80 2024-03-23T14:37:19.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.888+0000 7f11267fc640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f110806b810 0x7f110806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:19.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.889+0000 7f11267fc640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f110806b810 0x7f110806dcd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f11281558a0 tx=0x7f1118008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:19.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:19.892+0000 7f111ffff640 1 -- 172.21.15.161:0/1170174662 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1110076960 con 0x7f1114004a80 2024-03-23T14:37:20.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.102+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1128073570 con 0x7f1114004a80 2024-03-23T14:37:20.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.102+0000 7f111ffff640 1 -- 172.21.15.161:0/1170174662 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f11100480a0 con 0x7f1114004a80 2024-03-23T14:37:20.098 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:37:20.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f110806b810 msgr2=0x7f110806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f110806b810 0x7f110806dcd0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f11281558a0 tx=0x7f1118008040 comp rx=0 tx=0).stop 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 msgr2=0x7f1128152b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1128152b00 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f111002bf90 tx=0x7f1110004730 comp rx=0 tx=0).stop 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 shutdown_connections 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f110806b810 0x7f110806dcd0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f112813f040 0x7f112813f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1128150b80 0x7f11280b0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 --2- 172.21.15.161:0/1170174662 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1114004a80 0x7f1128152b00 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 >> 172.21.15.161:0/1170174662 conn(0x7f112806b4a0 msgr2=0x7f112815e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 shutdown_connections 2024-03-23T14:37:20.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.104+0000 7f112d981640 1 -- 172.21.15.161:0/1170174662 wait complete. 2024-03-23T14:37:20.111 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:37:20.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.224+0000 7f8fe72df640 1 Processor -- start 2024-03-23T14:37:20.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.225+0000 7f8fe72df640 1 -- start start 2024-03-23T14:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.225+0000 7f8fe72df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe0161820 0x7f8fe00d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.225+0000 7f8fe72df640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f8fe00d1270 0x7f8fe00d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.225+0000 7f8fe72df640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fe0161c00 con 0x7f8fe00d29d0 2024-03-23T14:37:20.224 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.225+0000 7f8fe72df640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f8fe00d2e20 con 0x7f8fe00d1270 2024-03-23T14:37:20.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.225+0000 7f8fe72df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe00cea30 con 0x7f8fe0161820 2024-03-23T14:37:20.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe587c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8fe00d29d0 0x7f8fe0161e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:44528/0 (socket says 172.21.15.161:44528) 2024-03-23T14:37:20.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe587c640 1 -- 172.21.15.161:0/4063518159 learned_addr learned my addr 172.21.15.161:0/4063518159 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:20.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe487a640 1 --2- 172.21.15.161:0/4063518159 >> v2:172.21.15.183:3300/0 conn(0x7f8fe00d1270 0x7f8fe00d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe507b640 1 --2- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe0161820 0x7f8fe00d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fd7fff640 1 -- 172.21.15.161:0/4063518159 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 979075677 0 0) 0x7f8fe0161c00 con 0x7f8fe00d29d0 2024-03-23T14:37:20.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fd7fff640 1 -- 172.21.15.161:0/4063518159 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fcc003520 con 0x7f8fe00d29d0 2024-03-23T14:37:20.225 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe487a640 1 -- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe0161820 msgr2=0x7f8fe00d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe487a640 1 --2- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe0161820 0x7f8fe00d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe487a640 1 -- 172.21.15.161:0/4063518159 >> v1:172.21.15.161:6789/0 conn(0x7f8fe00d29d0 legacy=0x7f8fe0161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe487a640 1 -- 172.21.15.161:0/4063518159 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fe00cfd70 con 0x7f8fe00d1270 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.226+0000 7f8fe507b640 1 --2- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe0161820 0x7f8fe00d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.227+0000 7f8fe487a640 1 --2- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe00d1270 0x7f8fe00d2490 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0097a0 tx=0x7f8fdc02b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f0602946b111aad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.227+0000 7f8fd7fff640 1 -- 172.21.15.161:0/4063518159 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc02d040 con 0x7f8fe00d1270 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.227+0000 7f8fd7fff640 1 -- 172.21.15.161:0/4063518159 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fdc031810 con 0x7f8fe00d1270 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.227+0000 7f8fd7fff640 1 -- 172.21.15.161:0/4063518159 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fdc031ba0 con 0x7f8fe00d1270 2024-03-23T14:37:20.226 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.228+0000 7f8fe72df640 1 -- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe00d1270 msgr2=0x7f8fe00d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.228+0000 7f8fe72df640 1 --2- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe00d1270 0x7f8fe00d2490 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc0097a0 tx=0x7f8fdc02b8b0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.228+0000 7f8fe72df640 1 -- 172.21.15.161:0/4063518159 shutdown_connections 2024-03-23T14:37:20.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.228+0000 7f8fe72df640 1 --2- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe00d1270 0x7f8fe00d2490 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.228+0000 7f8fe72df640 1 --2- 172.21.15.161:0/4063518159 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe0161820 0x7f8fe00d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.228+0000 7f8fe72df640 1 -- 172.21.15.161:0/4063518159 >> 172.21.15.161:0/4063518159 conn(0x7f8fe009d190 msgr2=0x7f8fe00d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:20.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.228+0000 7f8fe72df640 1 -- 172.21.15.161:0/4063518159 shutdown_connections 2024-03-23T14:37:20.227 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.228+0000 7f8fe72df640 1 -- 172.21.15.161:0/4063518159 wait complete. 2024-03-23T14:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.230+0000 7f8fe72df640 1 Processor -- start 2024-03-23T14:37:20.230 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.230+0000 7f8fe72df640 1 -- start start 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe72df640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe00d1270 0x7f8fe00cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe72df640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe00d29d0 0x7f8fe00c83f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe72df640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe0161820 0x7f8fe00cbce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe72df640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe00ce260 con 0x7f8fe00d1270 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe72df640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8fe00c8930 con 0x7f8fe0161820 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe72df640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8fe015c670 con 0x7f8fe00d29d0 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe00d1270 0x7f8fe00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe00d1270 0x7f8fe00cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:37900/0 (socket says 172.21.15.161:37900) 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 -- 172.21.15.161:0/1475059500 learned_addr learned my addr 172.21.15.161:0/1475059500 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe587c640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe0161820 0x7f8fe00cbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.231 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe487a640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe00d29d0 0x7f8fe00c83f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 -- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe00d29d0 msgr2=0x7f8fe00c83f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe00d29d0 0x7f8fe00c83f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 -- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe0161820 msgr2=0x7f8fe00cbce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe0161820 0x7f8fe00cbce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 -- 172.21.15.161:0/1475059500 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fd0009400 con 0x7f8fe00d1270 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.231+0000 7f8fe507b640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe00d1270 0x7f8fe00cdd20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000ebe0 tx=0x7f8fd000c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.232+0000 7f8fd5ffb640 1 -- 172.21.15.161:0/1475059500 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd0009760 con 0x7f8fe00d1270 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.232+0000 7f8fd5ffb640 1 -- 172.21.15.161:0/1475059500 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd000cc30 con 0x7f8fe00d1270 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.232+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fdc009400 con 0x7f8fe00d1270 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.232+0000 7f8fd5ffb640 1 -- 172.21.15.161:0/1475059500 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd000eee0 con 0x7f8fe00d1270 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.232+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fe00c9730 con 0x7f8fe00d1270 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.233+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fe015ac80 con 0x7f8fe00d1270 2024-03-23T14:37:20.232 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.233+0000 7f8fd5ffb640 1 -- 172.21.15.161:0/1475059500 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8fd0013470 con 0x7f8fe00d1270 2024-03-23T14:37:20.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.234+0000 7f8fd5ffb640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8fc406b810 0x7f8fc406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.234+0000 7f8fd5ffb640 1 -- 172.21.15.161:0/1475059500 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f8fd008a4a0 con 0x7f8fe00d1270 2024-03-23T14:37:20.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.234+0000 7f8fe487a640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8fc406b810 0x7f8fc406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.233 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.235+0000 7f8fe487a640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8fc406b810 0x7f8fc406dcd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc036000 tx=0x7f8fdc03c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:20.234 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.238+0000 7f8fd5ffb640 1 -- 172.21.15.161:0/1475059500 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8fd0054ed0 con 0x7f8fe00d1270 2024-03-23T14:37:20.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.430+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8fe004f4e0 con 0x7f8fe00d1270 2024-03-23T14:37:20.426 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.430+0000 7f8fd5ffb640 1 -- 172.21.15.161:0/1475059500 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7f8fd0058590 con 0x7f8fe00d1270 2024-03-23T14:37:20.427 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:37:20.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8fc406b810 msgr2=0x7f8fc406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8fc406b810 0x7f8fc406dcd0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc036000 tx=0x7f8fdc03c040 comp rx=0 tx=0).stop 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe00d1270 msgr2=0x7f8fe00cdd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe00d1270 0x7f8fe00cdd20 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f8fd000ebe0 tx=0x7f8fd000c520 comp rx=0 tx=0).stop 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 shutdown_connections 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8fc406b810 0x7f8fc406dcd0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8fe0161820 0x7f8fe00cbce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8fe00d29d0 0x7f8fe00c83f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 --2- 172.21.15.161:0/1475059500 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8fe00d1270 0x7f8fe00cdd20 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 >> 172.21.15.161:0/1475059500 conn(0x7f8fe009d190 msgr2=0x7f8fe006c030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 shutdown_connections 2024-03-23T14:37:20.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.432+0000 7f8fe72df640 1 -- 172.21.15.161:0/1475059500 wait complete. 2024-03-23T14:37:20.443 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:37:20.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.553+0000 7f8c9e084640 1 Processor -- start 2024-03-23T14:37:20.549 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.553+0000 7f8c9e084640 1 -- start start 2024-03-23T14:37:20.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c9e084640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c9815e400 0x7f8c980cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c9e084640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c980cf9d0 0x7f8c980d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c9e084640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c980d0460 con 0x7f8c980d1470 2024-03-23T14:37:20.551 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c9e084640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8c980d1050 con 0x7f8c980cf9d0 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c9e084640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8c980d06b0 con 0x7f8c9815e400 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c96ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c980cf9d0 0x7f8c980d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c96ffd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8c980cf9d0 0x7f8c980d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41788/0 (socket says 172.21.15.161:41788) 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c96ffd640 1 -- 172.21.15.161:0/2095738593 learned_addr learned my addr 172.21.15.161:0/2095738593 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c977fe640 1 --2- 172.21.15.161:0/2095738593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c9815e400 0x7f8c980cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c96ffd640 1 -- 172.21.15.161:0/2095738593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c9815e400 msgr2=0x7f8c980cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c96ffd640 1 --2- 172.21.15.161:0/2095738593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c9815e400 0x7f8c980cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c96ffd640 1 -- 172.21.15.161:0/2095738593 >> v1:172.21.15.183:6789/0 conn(0x7f8c980d1470 legacy=0x7f8c980b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c96ffd640 1 -- 172.21.15.161:0/2095738593 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c980d0990 con 0x7f8c980cf9d0 2024-03-23T14:37:20.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.555+0000 7f8c977fe640 1 --2- 172.21.15.161:0/2095738593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c9815e400 0x7f8c980cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:20.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.556+0000 7f8c96ffd640 1 --2- 172.21.15.161:0/2095738593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c980cf9d0 0x7f8c980d2430 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009870 tx=0x7f8c8c02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bd76bd10871de18 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:20.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.556+0000 7f8c967fc640 1 -- 172.21.15.161:0/2095738593 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c030840 con 0x7f8c980cf9d0 2024-03-23T14:37:20.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.556+0000 7f8c967fc640 1 -- 172.21.15.161:0/2095738593 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c8c0309a0 con 0x7f8c980cf9d0 2024-03-23T14:37:20.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.556+0000 7f8c967fc640 1 -- 172.21.15.161:0/2095738593 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c8c030d30 con 0x7f8c980cf9d0 2024-03-23T14:37:20.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.557+0000 7f8c9e084640 1 -- 172.21.15.161:0/2095738593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c980cf9d0 msgr2=0x7f8c980d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.557+0000 7f8c9e084640 1 --2- 172.21.15.161:0/2095738593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c980cf9d0 0x7f8c980d2430 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f8c8c009870 tx=0x7f8c8c02b980 comp rx=0 tx=0).stop 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.557+0000 7f8c9e084640 1 -- 172.21.15.161:0/2095738593 shutdown_connections 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.557+0000 7f8c9e084640 1 --2- 172.21.15.161:0/2095738593 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c980cf9d0 0x7f8c980d2430 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.557+0000 7f8c9e084640 1 --2- 172.21.15.161:0/2095738593 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c9815e400 0x7f8c980cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.557+0000 7f8c9e084640 1 -- 172.21.15.161:0/2095738593 >> 172.21.15.161:0/2095738593 conn(0x7f8c9809d190 msgr2=0x7f8c980d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.557+0000 7f8c9e084640 1 -- 172.21.15.161:0/2095738593 shutdown_connections 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.557+0000 7f8c9e084640 1 -- 172.21.15.161:0/2095738593 wait complete. 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.558+0000 7f8c9e084640 1 Processor -- start 2024-03-23T14:37:20.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.558+0000 7f8c9e084640 1 -- start start 2024-03-23T14:37:20.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c9e084640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c980cf9d0 0x7f8c980ca340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c9e084640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c980d1470 0x7f8c981f34b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c9e084640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c9815e400 0x7f8c980cb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c9e084640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8c980cbfa0 con 0x7f8c9815e400 2024-03-23T14:37:20.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c9e084640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8c980cc280 con 0x7f8c980d1470 2024-03-23T14:37:20.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c9e084640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8c980cc560 con 0x7f8c980cf9d0 2024-03-23T14:37:20.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c9815e400 0x7f8c980cb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c9815e400 0x7f8c980cb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41798/0 (socket says 172.21.15.161:41798) 2024-03-23T14:37:20.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 -- 172.21.15.161:0/3146368511 learned_addr learned my addr 172.21.15.161:0/3146368511 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:20.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c96ffd640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c980d1470 0x7f8c981f34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c977fe640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c980cf9d0 0x7f8c980ca340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 -- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c980cf9d0 msgr2=0x7f8c980ca340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c980cf9d0 0x7f8c980ca340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 -- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c980d1470 msgr2=0x7f8c981f34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c980d1470 0x7f8c981f34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 -- 172.21.15.161:0/3146368511 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c8c0094d0 con 0x7f8c9815e400 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c96ffd640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c980d1470 0x7f8c981f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.559+0000 7f8c97fff640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c9815e400 0x7f8c980cb8e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000c7a0 tx=0x7f8c9000cc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.560+0000 7f8c7ffff640 1 -- 172.21.15.161:0/3146368511 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90003b70 con 0x7f8c9815e400 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.560+0000 7f8c7ffff640 1 -- 172.21.15.161:0/3146368511 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c90003cf0 con 0x7f8c9815e400 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.560+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c980cc840 con 0x7f8c9815e400 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.560+0000 7f8c7ffff640 1 -- 172.21.15.161:0/3146368511 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90003ea0 con 0x7f8c9815e400 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.560+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c98159830 con 0x7f8c9815e400 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.561+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c980ccd60 con 0x7f8c9815e400 2024-03-23T14:37:20.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.561+0000 7f8c7ffff640 1 -- 172.21.15.161:0/3146368511 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8c9000aa90 con 0x7f8c9815e400 2024-03-23T14:37:20.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.562+0000 7f8c7ffff640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c8406b810 0x7f8c8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:20.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.562+0000 7f8c7ffff640 1 -- 172.21.15.161:0/3146368511 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f8c9008a040 con 0x7f8c9815e400 2024-03-23T14:37:20.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.562+0000 7f8c977fe640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c8406b810 0x7f8c8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:20.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.563+0000 7f8c977fe640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c8406b810 0x7f8c8406dcd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8c880096d0 tx=0x7f8c88008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:20.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.566+0000 7f8c7ffff640 1 -- 172.21.15.161:0/3146368511 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8c980ccd60 con 0x7f8c9815e400 2024-03-23T14:37:20.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.756+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f8c980ccd60 con 0x7f8c9815e400 2024-03-23T14:37:20.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.756+0000 7f8c7ffff640 1 -- 172.21.15.161:0/3146368511 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) ==== 80+0+0 (secure 0 0 0) 0x7f8c980ccd60 con 0x7f8c9815e400 2024-03-23T14:37:20.752 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:37:20.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c8406b810 msgr2=0x7f8c8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c8406b810 0x7f8c8406dcd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f8c880096d0 tx=0x7f8c88008040 comp rx=0 tx=0).stop 2024-03-23T14:37:20.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c9815e400 msgr2=0x7f8c980cb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:20.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c9815e400 0x7f8c980cb8e0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000c7a0 tx=0x7f8c9000cc60 comp rx=0 tx=0).stop 2024-03-23T14:37:20.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 shutdown_connections 2024-03-23T14:37:20.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8c8406b810 0x7f8c8406dcd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8c9815e400 0x7f8c980cb8e0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8c980d1470 0x7f8c981f34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 --2- 172.21.15.161:0/3146368511 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8c980cf9d0 0x7f8c980ca340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:20.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 >> 172.21.15.161:0/3146368511 conn(0x7f8c9809d190 msgr2=0x7f8c980b22d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:20.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 shutdown_connections 2024-03-23T14:37:20.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:20.758+0000 7f8c9e084640 1 -- 172.21.15.161:0/3146368511 wait complete. 2024-03-23T14:37:20.765 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-23T14:37:20.765 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:20.765 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:20.791 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[57973]: fuse finished with error 0 and tester_r 0 2024-03-23T14:37:20.792 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:37:26.894 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:37:26.894 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:26.895 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T14:37:26.910 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-03-23T14:37:27.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.056+0000 7fd77e53a640 1 Processor -- start 2024-03-23T14:37:27.052 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.056+0000 7fd77e53a640 1 -- start start 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.057+0000 7fd77e53a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7780d2b40 0x7fd7780b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.057+0000 7fd77e53a640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7780d0460 con 0x7fd7780d1470 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.057+0000 7fd77e53a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7780d06b0 con 0x7fd77815e400 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.057+0000 7fd77e53a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd7780d1050 con 0x7fd7780d2b40 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.057+0000 7fd7777fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fd7780d1470 0x7fd7780d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35482/0 (socket says 172.21.15.161:35482) 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.057+0000 7fd7777fe640 1 -- 172.21.15.161:0/3510265910 learned_addr learned my addr 172.21.15.161:0/3510265910 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd77cad7640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7780d2b40 0x7fd7780b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 569243365 0 0) 0x7fd7780d0460 con 0x7fd7780d1470 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd764003520 con 0x7fd7780d1470 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1216895344 0 0) 0x7fd764003520 con 0x7fd7780d1470 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7780d2b40 msgr2=0x7fd7780b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:27.056 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7780d2b40 0x7fd7780b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 >> v1:172.21.15.183:6789/0 conn(0x7fd77815e400 legacy=0x7fd7780cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7780d0990 con 0x7fd7780d1470 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fd768001670 con 0x7fd7780d1470 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 >> v1:172.21.15.161:6789/0 conn(0x7fd7780d1470 legacy=0x7fd7780d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7640039a0 0x7fd764003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7640047c0 0x7fd764004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7780d2b40 0x7fd764005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd7780d1050 con 0x7fd7640047c0 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd76400ba60 con 0x7fd7640039a0 2024-03-23T14:37:27.057 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd76400bc20 con 0x7fd7780d2b40 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd7777fe640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7640047c0 0x7fd764004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd777fff640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7640039a0 0x7fd764003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd77cad7640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7780d2b40 0x7fd764005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd7777fe640 1 -- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7780d2b40 msgr2=0x7fd764005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd7777fe640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7780d2b40 0x7fd764005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd7777fe640 1 -- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7640039a0 msgr2=0x7fd764003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd7777fe640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7640039a0 0x7fd764003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd7777fe640 1 -- 172.21.15.161:0/3510265910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd764003520 con 0x7fd7640047c0 2024-03-23T14:37:27.058 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.058+0000 7fd777fff640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7640039a0 0x7fd764003dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.059+0000 7fd7777fe640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7640047c0 0x7fd764004c40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd76800c210 tx=0x7fd7640057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d99d956a8b16cb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.059+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd768005ad0 con 0x7fd7640047c0 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.059+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd768005c50 con 0x7fd7640047c0 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.059+0000 7fd776ffd640 1 -- 172.21.15.161:0/3510265910 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76802c870 con 0x7fd7640047c0 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.060+0000 7fd77e53a640 1 -- 172.21.15.161:0/3510265910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7640047c0 msgr2=0x7fd764004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.060+0000 7fd77e53a640 1 --2- 172.21.15.161:0/3510265910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd7640047c0 0x7fd764004c40 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7fd76800c210 tx=0x7fd7640057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.060+0000 7fd77cad7640 1 -- 172.21.15.161:0/3510265910 reap_dead start 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.060+0000 7fd77e53a640 1 -- 172.21.15.161:0/3510265910 shutdown_connections 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.060+0000 7fd77e53a640 1 -- 172.21.15.161:0/3510265910 >> 172.21.15.161:0/3510265910 conn(0x7fd77809d190 msgr2=0x7fd7780d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.060+0000 7fd77e53a640 1 -- 172.21.15.161:0/3510265910 shutdown_connections 2024-03-23T14:37:27.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.060+0000 7fd77e53a640 1 -- 172.21.15.161:0/3510265910 wait complete. 2024-03-23T14:37:27.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.061+0000 7fd77e53a640 1 Processor -- start 2024-03-23T14:37:27.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.061+0000 7fd77e53a640 1 -- start start 2024-03-23T14:37:27.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd77e53a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7640047c0 0x7fd77814f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:27.062 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd77e53a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd778161560 0x7fd77815fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd77e53a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7780cb2b0 0x7fd7780b8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd77e53a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fd77814f850 con 0x7fd778161560 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd77e53a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fd7781610d0 con 0x7fd7780cb2b0 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd77e53a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fd7781604c0 con 0x7fd7640047c0 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd778161560 0x7fd77815fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd778161560 0x7fd77815fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41802/0 (socket says 172.21.15.161:41802) 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 -- 172.21.15.161:0/1249216316 learned_addr learned my addr 172.21.15.161:0/1249216316 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd777fff640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7640047c0 0x7fd77814f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd77cad7640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7780cb2b0 0x7fd7780b8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 -- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7640047c0 msgr2=0x7fd77814f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7640047c0 0x7fd77814f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 -- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7780cb2b0 msgr2=0x7fd7780b8bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7780cb2b0 0x7fd7780b8bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 -- 172.21.15.161:0/1249216316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7781607a0 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.063+0000 7fd7777fe640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd778161560 0x7fd77815fbe0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd76800c330 tx=0x7fd768001a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.064+0000 7fd774ff9640 1 -- 172.21.15.161:0/1249216316 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd76802cc80 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.064+0000 7fd774ff9640 1 -- 172.21.15.161:0/1249216316 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd76802cde0 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.064+0000 7fd774ff9640 1 -- 172.21.15.161:0/1249216316 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7680072b0 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.064+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd778160a80 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.064+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd778160120 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.065+0000 7fd774ff9640 1 -- 172.21.15.161:0/1249216316 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fd768032070 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.065+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7780ce270 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.065+0000 7fd774ff9640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd75806b810 0x7fd75806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.065+0000 7fd774ff9640 1 -- 172.21.15.161:0/1249216316 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fd76803c040 con 0x7fd778161560 2024-03-23T14:37:27.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.066+0000 7fd777fff640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd75806b810 0x7fd75806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:27.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.066+0000 7fd777fff640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd75806b810 0x7fd75806dcd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd760007fd0 tx=0x7fd760009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:27.065 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.070+0000 7fd774ff9640 1 -- 172.21.15.161:0/1249216316 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fd7680761d0 con 0x7fd778161560 2024-03-23T14:37:27.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.303+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fd7780a1e00 con 0x7fd778161560 2024-03-23T14:37:27.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.303+0000 7fd774ff9640 1 -- 172.21.15.161:0/1249216316 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7fd768079890 con 0x7fd778161560 2024-03-23T14:37:27.299 INFO:teuthology.orchestra.run.smithi161.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-23T14:37:27.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd75806b810 msgr2=0x7fd75806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:27.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd75806b810 0x7fd75806dcd0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fd760007fd0 tx=0x7fd760009040 comp rx=0 tx=0).stop 2024-03-23T14:37:27.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd778161560 msgr2=0x7fd77815fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd778161560 0x7fd77815fbe0 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7fd76800c330 tx=0x7fd768001a00 comp rx=0 tx=0).stop 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 shutdown_connections 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fd75806b810 0x7fd75806dcd0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fd7780cb2b0 0x7fd7780b8bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fd778161560 0x7fd77815fbe0 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 --2- 172.21.15.161:0/1249216316 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fd7640047c0 0x7fd77814f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 >> 172.21.15.161:0/1249216316 conn(0x7fd77809d190 msgr2=0x7fd7780b20f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 shutdown_connections 2024-03-23T14:37:27.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27.306+0000 7fd77e53a640 1 -- 172.21.15.161:0/1249216316 wait complete. 2024-03-23T14:37:27.313 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-23T14:37:27.313 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-23T14:37:27.313 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi161.front.sepia.ceph.com 2024-03-23T14:37:27.314 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-23T14:37:27.314 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:27.314 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-23T14:37:27.314 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-23T14:37:27.314 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-23T14:37:27.314 INFO:tasks.cephfs.mount:Setting the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:27.314 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:27.315 DEBUG:teuthology.orchestra.run.smithi161:> ip addr 2024-03-23T14:37:27.330 INFO:teuthology.orchestra.run.smithi161.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-23T14:37:27.330 INFO:teuthology.orchestra.run.smithi161.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-23T14:37:27.330 INFO:teuthology.orchestra.run.smithi161.stdout: inet 127.0.0.1/8 scope host lo 2024-03-23T14:37:27.330 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:27.330 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 ::1/128 scope host 2024-03-23T14:37:27.330 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7c brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f0 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:d9:36:7d brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: altname enp7s0f1 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:50 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0c:c4:7a:88:72:51 brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: inet 172.21.15.161/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft 42623sec preferred_lft 42623sec 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::ec4:7aff:fe88:7251/64 scope link 2024-03-23T14:37:27.331 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:6: ceph-brx: mtu 1500 qdisc noqueue state UP group default qlen 1000 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: inet 192.168.255.254/16 brd 192.168.255.255 scope global ceph-brx 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c4a9:64ff:fe59:877a/64 scope link 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:7: brx.0@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 0e:48:50:01:9f:4e brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::c48:50ff:fe01:9f4e/64 scope link 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout:8: brx.1@if2: mtu 1500 qdisc noqueue master ceph-brx state UP group default qlen 1000 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: link/ether 62:02:d5:17:e7:0d brd ff:ff:ff:ff:ff:ff link-netns ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:27.332 INFO:teuthology.orchestra.run.smithi161.stdout: inet6 fe80::6002:d5ff:fe17:e70d/64 scope link 2024-03-23T14:37:27.333 INFO:teuthology.orchestra.run.smithi161.stdout: valid_lft forever preferred_lft forever 2024-03-23T14:37:27.333 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:27.333 DEBUG:teuthology.orchestra.run.smithi161:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-23T14:37:27.401 INFO:teuthology.orchestra.run.smithi161.stdout:1 2024-03-23T14:37:27.402 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:27.403 DEBUG:teuthology.orchestra.run.smithi161:> route 2024-03-23T14:37:27.464 INFO:teuthology.orchestra.run.smithi161.stdout:Kernel IP routing table 2024-03-23T14:37:27.464 INFO:teuthology.orchestra.run.smithi161.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-23T14:37:27.464 INFO:teuthology.orchestra.run.smithi161.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-23T14:37:27.464 INFO:teuthology.orchestra.run.smithi161.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-23T14:37:27.465 INFO:teuthology.orchestra.run.smithi161.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-23T14:37:27.465 DEBUG:teuthology.orchestra.run.smithi161:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-03-23T14:37:27.465 DEBUG:teuthology.orchestra.run.smithi161:> set -e 2024-03-23T14:37:27.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-23T14:37:27.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-23T14:37:27.465 DEBUG:teuthology.orchestra.run.smithi161:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-23T14:37:27.465 DEBUG:teuthology.orchestra.run.smithi161:> ') 2024-03-23T14:37:27.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-03-23T14:37:27.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:27 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:37:27.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:27.601 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:37:27.617 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T14:37:27.617 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T14:37:27.617 INFO:tasks.cephfs.mount:Resuming the 'ceph-ns--home-ubuntu-cephtest-mnt.0' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:27.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:27.617 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link set brx.0 up 2024-03-23T14:37:27.686 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-03-23T14:37:27.686 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:37:27.686 DEBUG:teuthology.orchestra.run.smithi161:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:27.741 INFO:teuthology.orchestra.run.smithi161.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-23T14:37:27.742 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-23T14:37:27.742 DEBUG:teuthology.orchestra.run.smithi161:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:27.796 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:37:27.865 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:37:27.922 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:37:27.922 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:37:27.922 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:37:27.922 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:37:27.922 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:37:27.922 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:37:27.922 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:37:27.923 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:37:27.924 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:37:27.925 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:37:27.925 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:37:27.925 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:37:27.925 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:37:27.925 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:37:27.925 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:37:27.925 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:37:27.926 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:37:27.926 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:37:27.926 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:27.926 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:27.926 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:27.926 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:27.927 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:27.927 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:37:27.987 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-23T14:37:27.987 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-03-23T14:37:28.035 DEBUG:teuthology.orchestra.run.smithi161:> sudo modprobe fuse 2024-03-23T14:37:28.076 DEBUG:teuthology.orchestra.run.smithi161:> cat /proc/self/mounts | awk '{print $2}' 2024-03-23T14:37:28.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:2024-03-23T14:37:28.125+0000 7f6c7d04b480 -1 init, newargv = 0x55e1accd7c20 newargc=15 2024-03-23T14:37:28.121 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[58938]: starting ceph client 2024-03-23T14:37:28.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[58938]: starting fuse 2024-03-23T14:37:28.141 INFO:teuthology.orchestra.run.smithi161.stdout:/proc 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/sys 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/dev 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/security 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/shm 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/pts 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/run 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/cgroup 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/pstore 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/bpf 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/ 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/selinux 2024-03-23T14:37:28.142 INFO:teuthology.orchestra.run.smithi161.stdout:/proc/sys/fs/binfmt_misc 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/debug 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/mqueue 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/dev/hugepages 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/tracing 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/fs/fuse/connections 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/sys/kernel/config 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-sysctl.service 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/nfs/rpc_pipefs 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/run/user/1000 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-0 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-1 2024-03-23T14:37:28.143 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-2 2024-03-23T14:37:28.144 INFO:teuthology.orchestra.run.smithi161.stdout:/var/lib/ceph/osd/ceph-3 2024-03-23T14:37:28.144 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns 2024-03-23T14:37:28.144 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:28.144 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:28.144 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:28.144 INFO:teuthology.orchestra.run.smithi161.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:28.144 INFO:teuthology.orchestra.run.smithi161.stdout:/home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:28.144 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:28.145 DEBUG:teuthology.orchestra.run.smithi161:> ls /sys/fs/fuse/connections 2024-03-23T14:37:28.204 INFO:teuthology.orchestra.run.smithi161.stdout:41 2024-03-23T14:37:28.204 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> sudo stdin-killer -- python3 -c ' 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> import glob 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> import re 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> import subprocess 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> def _find_admin_socket(client_name): 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> files = glob.glob(asok_path) 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:28.205 DEBUG:teuthology.orchestra.run.smithi161:> # Given a non-glob path, it better be there 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> if "*" not in asok_path: 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> assert(len(files) == 1) 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> return files[0] 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> for f in files: 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> contents = proc_f.read() 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> if mountpoint in contents: 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> return f 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:28.206 DEBUG:teuthology.orchestra.run.smithi161:> print(_find_admin_socket("client.0")) 2024-03-23T14:37:28.207 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:37:28.330 INFO:teuthology.orchestra.run.smithi161.stdout:/var/run/ceph/ceph-client.0.58938.asok 2024-03-23T14:37:28.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-23T14:37:28.346 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.58938.asok 2024-03-23T14:37:28.346 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:28.346 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.58938.asok status 2024-03-23T14:37:28.504 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:28.504 INFO:teuthology.orchestra.run.smithi161.stdout: "metadata": { 2024-03-23T14:37:28.504 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_sha1": "21b3bd672163092b4e542f772797fcba02d3d4de", 2024-03-23T14:37:28.505 INFO:teuthology.orchestra.run.smithi161.stdout: "ceph_version": "ceph version 19.0.0-1855-g21b3bd67 (21b3bd672163092b4e542f772797fcba02d3d4de) squid (dev)", 2024-03-23T14:37:28.505 INFO:teuthology.orchestra.run.smithi161.stdout: "entity_id": "0", 2024-03-23T14:37:28.505 INFO:teuthology.orchestra.run.smithi161.stdout: "hostname": "smithi161", 2024-03-23T14:37:28.505 INFO:teuthology.orchestra.run.smithi161.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-03-23T14:37:28.505 INFO:teuthology.orchestra.run.smithi161.stdout: "pid": "58938", 2024-03-23T14:37:28.505 INFO:teuthology.orchestra.run.smithi161.stdout: "root": "/" 2024-03-23T14:37:28.505 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:28.506 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_count": 0, 2024-03-23T14:37:28.506 INFO:teuthology.orchestra.run.smithi161.stdout: "dentry_pinned_count": 0, 2024-03-23T14:37:28.506 INFO:teuthology.orchestra.run.smithi161.stdout: "id": 6249, 2024-03-23T14:37:28.506 INFO:teuthology.orchestra.run.smithi161.stdout: "inst": { 2024-03-23T14:37:28.506 INFO:teuthology.orchestra.run.smithi161.stdout: "name": { 2024-03-23T14:37:28.506 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "client", 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "num": 6249 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 4268497042 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: } 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": { 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "type": "v1", 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "addr": "192.168.0.1:0", 2024-03-23T14:37:28.507 INFO:teuthology.orchestra.run.smithi161.stdout: "nonce": 4268497042 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: }, 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: "inst_str": "client.6249 192.168.0.1:0/4268497042", 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: "addr_str": "192.168.0.1:0/4268497042", 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: "inode_count": 1, 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: "mds_epoch": 23, 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch": 48, 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: "osd_epoch_barrier": 0, 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: "blocklisted": false, 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout: "fs_name": "cephfs" 2024-03-23T14:37:28.508 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:28.513 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:28.513 DEBUG:teuthology.orchestra.run.smithi161:> stat --file-system '--printf=%T 2024-03-23T14:37:28.513 DEBUG:teuthology.orchestra.run.smithi161:> ' -- /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:28.535 INFO:teuthology.orchestra.run.smithi161.stdout:fuseblk 2024-03-23T14:37:28.535 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:28.535 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:28.535 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:28.608 DEBUG:teuthology.orchestra.run.smithi161:> stdin-killer -- python3 -c ' 2024-03-23T14:37:28.608 DEBUG:teuthology.orchestra.run.smithi161:> import os 2024-03-23T14:37:28.608 DEBUG:teuthology.orchestra.run.smithi161:> import errno 2024-03-23T14:37:28.608 DEBUG:teuthology.orchestra.run.smithi161:> 2024-03-23T14:37:28.608 DEBUG:teuthology.orchestra.run.smithi161:> fd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR) 2024-03-23T14:37:28.608 DEBUG:teuthology.orchestra.run.smithi161:> try: 2024-03-23T14:37:28.608 DEBUG:teuthology.orchestra.run.smithi161:> if True: 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> ret = os.read(fd, 1024) 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> else: 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> os.write(fd, b'"'"'content'"'"') 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> except OSError as e: 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> if e.errno != errno.EPERM: 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> raise 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> else: 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> raise RuntimeError("client does not check permission of data pool") 2024-03-23T14:37:28.609 DEBUG:teuthology.orchestra.run.smithi161:> ' 2024-03-23T14:37:28.717 INFO:teuthology.orchestra.run.smithi161.stderr:Traceback (most recent call last): 2024-03-23T14:37:28.717 INFO:teuthology.orchestra.run.smithi161.stderr: File "", line 15, in 2024-03-23T14:37:28.717 INFO:teuthology.orchestra.run.smithi161.stderr:RuntimeError: client does not check permission of data pool 2024-03-23T14:37:28.721 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28 stdin-killer INFO: command exited with status 1: exiting normally with same code! 2024-03-23T14:37:28.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:37:28.728 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2024-03-23T14:37:28.878 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.881+0000 7f18d298a640 1 Processor -- start 2024-03-23T14:37:28.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.881+0000 7f18d298a640 1 -- start start 2024-03-23T14:37:28.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d298a640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f18cc15e440 0x7f18cc0d4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:28.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d298a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0d24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:28.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d298a640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18cc0d2a20 0x7f18cc0ceaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:28.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d298a640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f18cc0d2eb0 con 0x7f18cc0d2a20 2024-03-23T14:37:28.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d298a640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f18cc0b0ac0 con 0x7f18cc15e440 2024-03-23T14:37:28.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d298a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f18cc0b4220 con 0x7f18cc0d1d50 2024-03-23T14:37:28.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d0f27640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18cc0d2a20 0x7f18cc0ceaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:28.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d0f27640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f18cc0d2a20 0x7f18cc0ceaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41818/0 (socket says 172.21.15.161:41818) 2024-03-23T14:37:28.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.882+0000 7f18d0f27640 1 -- 172.21.15.161:0/4252782444 learned_addr learned my addr 172.21.15.161:0/4252782444 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18cbfff640 1 --2- 172.21.15.161:0/4252782444 >> v2:172.21.15.183:3300/0 conn(0x7f18cc15e440 0x7f18cc0d4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18cb7fe640 1 --2- 172.21.15.161:0/4252782444 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0d24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18d0f27640 1 -- 172.21.15.161:0/4252782444 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 msgr2=0x7f18cc0d24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18d0f27640 1 --2- 172.21.15.161:0/4252782444 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0d24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18d0f27640 1 -- 172.21.15.161:0/4252782444 >> v2:172.21.15.183:3300/0 conn(0x7f18cc15e440 msgr2=0x7f18cc0d4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18d0f27640 1 --2- 172.21.15.161:0/4252782444 >> v2:172.21.15.183:3300/0 conn(0x7f18cc15e440 0x7f18cc0d4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18d0f27640 1 -- 172.21.15.161:0/4252782444 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18cc0b3350 con 0x7f18cc0d2a20 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18cbfff640 1 --2- 172.21.15.161:0/4252782444 >> v2:172.21.15.183:3300/0 conn(0x7f18cc15e440 0x7f18cc0d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:28.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.883+0000 7f18cb7fe640 1 --2- 172.21.15.161:0/4252782444 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:28.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d0f27640 1 --2- 172.21.15.161:0/4252782444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc0ceaa0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f18cc08e5c0 tx=0x7f18c402bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=235c4360ef2682f6 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:28.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18caffd640 1 -- 172.21.15.161:0/4252782444 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c402ca70 con 0x7f18cc0d2a20 2024-03-23T14:37:28.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18caffd640 1 -- 172.21.15.161:0/4252782444 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c402cbd0 con 0x7f18cc0d2a20 2024-03-23T14:37:28.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18caffd640 1 -- 172.21.15.161:0/4252782444 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c40343f0 con 0x7f18cc0d2a20 2024-03-23T14:37:28.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 -- 172.21.15.161:0/4252782444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 msgr2=0x7f18cc0ceaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:28.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 --2- 172.21.15.161:0/4252782444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc0ceaa0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f18cc08e5c0 tx=0x7f18c402bed0 comp rx=0 tx=0).stop 2024-03-23T14:37:28.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 -- 172.21.15.161:0/4252782444 shutdown_connections 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 --2- 172.21.15.161:0/4252782444 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc0ceaa0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 --2- 172.21.15.161:0/4252782444 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0d24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 --2- 172.21.15.161:0/4252782444 >> v2:172.21.15.183:3300/0 conn(0x7f18cc15e440 0x7f18cc0d4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 -- 172.21.15.161:0/4252782444 >> 172.21.15.161:0/4252782444 conn(0x7f18cc0d3f00 msgr2=0x7f18cc0d4310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 -- 172.21.15.161:0/4252782444 shutdown_connections 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.884+0000 7f18d298a640 1 -- 172.21.15.161:0/4252782444 wait complete. 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.885+0000 7f18d298a640 1 Processor -- start 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18d298a640 1 -- start start 2024-03-23T14:37:28.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18d298a640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0a0ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18d298a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc151590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18d298a640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18cc15e440 0x7f18cc09dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18d298a640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc0a1130 con 0x7f18cc0d2a20 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18d298a640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc0a1410 con 0x7f18cc15e440 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18d298a640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f18cc0a16f0 con 0x7f18cc0d1d50 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18cb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc151590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.886+0000 7f18cb7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc151590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41830/0 (socket says 172.21.15.161:41830) 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18cb7fe640 1 -- 172.21.15.161:0/3724414607 learned_addr learned my addr 172.21.15.161:0/3724414607 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:28.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18d0f27640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18cc15e440 0x7f18cc09dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18cbfff640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0a0ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18cb7fe640 1 -- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 msgr2=0x7f18cc0a0ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18cb7fe640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0a0ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18cb7fe640 1 -- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18cc15e440 msgr2=0x7f18cc09dc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18cb7fe640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18cc15e440 0x7f18cc09dc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18cb7fe640 1 -- 172.21.15.161:0/3724414607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18c4009b30 con 0x7f18cc0d2a20 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18cb7fe640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc151590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f18c000d7d0 tx=0x7f18c000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18c8ff9640 1 -- 172.21.15.161:0/3724414607 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c0004030 con 0x7f18cc0d2a20 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18c8ff9640 1 -- 172.21.15.161:0/3724414607 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c0004190 con 0x7f18cc0d2a20 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18cc0a19d0 con 0x7f18cc0d2a20 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18c8ff9640 1 -- 172.21.15.161:0/3724414607 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c0004520 con 0x7f18cc0d2a20 2024-03-23T14:37:28.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.887+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18cc1606a0 con 0x7f18cc0d2a20 2024-03-23T14:37:28.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.888+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18cc0a1bc0 con 0x7f18cc0d2a20 2024-03-23T14:37:28.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.889+0000 7f18c8ff9640 1 -- 172.21.15.161:0/3724414607 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f18c00096a0 con 0x7f18cc0d2a20 2024-03-23T14:37:28.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.889+0000 7f18c8ff9640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18b806b810 0x7f18b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:28.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.889+0000 7f18c8ff9640 1 -- 172.21.15.161:0/3724414607 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f18c008a780 con 0x7f18cc0d2a20 2024-03-23T14:37:28.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.889+0000 7f18cbfff640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18b806b810 0x7f18b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:28.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.890+0000 7f18cbfff640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18b806b810 0x7f18b806dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:28.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:28.898+0000 7f18c8ff9640 1 -- 172.21.15.161:0/3724414607 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f18c0055250 con 0x7f18cc0d2a20 2024-03-23T14:37:29.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.142+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) -- 0x7f18cc068160 con 0x7f18cc0d2a20 2024-03-23T14:37:29.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.144+0000 7f18c8ff9640 1 -- 172.21.15.161:0/3724414607 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v18) ==== 184+0+141 (secure 0 0 0) 0x7f18c000b380 con 0x7f18cc0d2a20 2024-03-23T14:37:29.140 INFO:teuthology.orchestra.run.smithi161.stdout:[client.0] 2024-03-23T14:37:29.141 INFO:teuthology.orchestra.run.smithi161.stdout: key = AQBh6P5lYkDZLRAAz1USeFu4uh8GW63mHIgW9g== 2024-03-23T14:37:29.141 INFO:teuthology.orchestra.run.smithi161.stdout: caps mds = "allow" 2024-03-23T14:37:29.141 INFO:teuthology.orchestra.run.smithi161.stdout: caps mon = "allow r" 2024-03-23T14:37:29.141 INFO:teuthology.orchestra.run.smithi161.stdout: caps osd = "allow rw pool=cephfs_data" 2024-03-23T14:37:29.141 INFO:teuthology.orchestra.run.smithi161.stderr:updated caps for client.0 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.146+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18b806b810 msgr2=0x7f18b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.146+0000 7f18d298a640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18b806b810 0x7f18b806dcd0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f18bc006fd0 tx=0x7f18bc008040 comp rx=0 tx=0).stop 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.146+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 msgr2=0x7f18cc151590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.146+0000 7f18d298a640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc151590 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f18c000d7d0 tx=0x7f18c000dc90 comp rx=0 tx=0).stop 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.147+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 shutdown_connections 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.147+0000 7f18d298a640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f18b806b810 0x7f18b806dcd0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.147+0000 7f18d298a640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f18cc15e440 0x7f18cc09dc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.147+0000 7f18d298a640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f18cc0d2a20 0x7f18cc151590 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.143 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.147+0000 7f18d298a640 1 --2- 172.21.15.161:0/3724414607 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f18cc0d1d50 0x7f18cc0a0ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.147+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 >> 172.21.15.161:0/3724414607 conn(0x7f18cc0d3f00 msgr2=0x7f18cc0b45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:29.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.147+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 shutdown_connections 2024-03-23T14:37:29.144 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.147+0000 7f18d298a640 1 -- 172.21.15.161:0/3724414607 wait complete. 2024-03-23T14:37:29.155 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2024-03-23T14:37:29.155 DEBUG:teuthology.orchestra.run.smithi161:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T14:37:29.194 DEBUG:teuthology.orchestra.run.smithi183:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2024-03-23T14:37:29.240 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2024-03-23T14:37:29.240 INFO:tasks.cephfs.mount:Terminating background process 2024-03-23T14:37:29.240 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:37:29.240 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:29.257 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:29.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.407+0000 7ff8e893e640 1 Processor -- start 2024-03-23T14:37:29.404 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.408+0000 7ff8e893e640 1 -- start start 2024-03-23T14:37:29.407 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.408+0000 7ff8e893e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8e40d2b40 0x7ff8e40b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.408+0000 7ff8e893e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8e40d0460 con 0x7ff8e415e400 2024-03-23T14:37:29.408 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.408+0000 7ff8e893e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8e40d06b0 con 0x7ff8e40d1470 2024-03-23T14:37:29.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.408+0000 7ff8e893e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e40d1050 con 0x7ff8e40d2b40 2024-03-23T14:37:29.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e259c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7ff8e415e400 0x7ff8e40cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35496/0 (socket says 172.21.15.161:35496) 2024-03-23T14:37:29.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e259c640 1 -- 172.21.15.161:0/3637116157 learned_addr learned my addr 172.21.15.161:0/3637116157 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:29.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3270496302 0 0) 0x7ff8e40d0460 con 0x7ff8e415e400 2024-03-23T14:37:29.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8cc003520 con 0x7ff8e415e400 2024-03-23T14:37:29.410 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e2d9d640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8e40d2b40 0x7ff8e40b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1697295301 0 0) 0x7ff8cc003520 con 0x7ff8e415e400 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8e40d2b40 msgr2=0x7ff8e40b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8e40d2b40 0x7ff8e40b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 >> v1:172.21.15.183:6789/0 conn(0x7ff8e40d1470 legacy=0x7ff8e40d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e40d0990 con 0x7ff8e415e400 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7ff8d4001670 con 0x7ff8e415e400 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 >> v1:172.21.15.161:6789/0 conn(0x7ff8e415e400 legacy=0x7ff8e40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8e40d2b40 0x7ff8cc0039a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8cc003fd0 0x7ff8cc0047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8cc004d00 0x7ff8cc005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e40d1050 con 0x7ff8cc004d00 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff8cc00bb40 con 0x7ff8cc003fd0 2024-03-23T14:37:29.411 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff8cc00bd00 con 0x7ff8e40d2b40 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e2d9d640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8cc004d00 0x7ff8cc005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e259c640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8e40d2b40 0x7ff8cc0039a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.409+0000 7ff8e1d9b640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8cc003fd0 0x7ff8cc0047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e2d9d640 1 -- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8e40d2b40 msgr2=0x7ff8cc0039a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e2d9d640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8e40d2b40 0x7ff8cc0039a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e2d9d640 1 -- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8cc003fd0 msgr2=0x7ff8cc0047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e2d9d640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8cc003fd0 0x7ff8cc0047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e2d9d640 1 -- 172.21.15.161:0/3637116157 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8cc003520 con 0x7ff8cc004d00 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e2d9d640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8cc004d00 0x7ff8cc005180 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc007360 tx=0x7ff8cc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74330cd53654b5a7 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc02ccd0 con 0x7ff8cc004d00 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8dc02ce30 con 0x7ff8cc004d00 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.410+0000 7ff8e159a640 1 -- 172.21.15.161:0/3637116157 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc034660 con 0x7ff8cc004d00 2024-03-23T14:37:29.412 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.411+0000 7ff8e893e640 1 -- 172.21.15.161:0/3637116157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8cc004d00 msgr2=0x7ff8cc005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.411+0000 7ff8e893e640 1 --2- 172.21.15.161:0/3637116157 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8cc004d00 0x7ff8cc005180 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc007360 tx=0x7ff8cc0057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.411+0000 7ff8e2d9d640 1 -- 172.21.15.161:0/3637116157 reap_dead start 2024-03-23T14:37:29.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.411+0000 7ff8e893e640 1 -- 172.21.15.161:0/3637116157 shutdown_connections 2024-03-23T14:37:29.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.411+0000 7ff8e893e640 1 -- 172.21.15.161:0/3637116157 >> 172.21.15.161:0/3637116157 conn(0x7ff8e409d190 msgr2=0x7ff8e40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:29.413 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.411+0000 7ff8e893e640 1 -- 172.21.15.161:0/3637116157 shutdown_connections 2024-03-23T14:37:29.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.411+0000 7ff8e893e640 1 -- 172.21.15.161:0/3637116157 wait complete. 2024-03-23T14:37:29.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.413+0000 7ff8e893e640 1 Processor -- start 2024-03-23T14:37:29.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.413+0000 7ff8e893e640 1 -- start start 2024-03-23T14:37:29.416 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e893e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8cc004d00 0x7ff8e40b97c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e893e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8e4159ee0 0x7ff8e40caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e893e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8e41601b0 0x7ff8e4160630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e893e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e4160b70 con 0x7ff8e41601b0 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e893e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e415a340 con 0x7ff8e4159ee0 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e893e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7ff8e40b9590 con 0x7ff8cc004d00 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8e41601b0 0x7ff8e4160630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e2d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8e41601b0 0x7ff8e4160630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41840/0 (socket says 172.21.15.161:41840) 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.414+0000 7ff8e2d9d640 1 -- 172.21.15.161:0/771134566 learned_addr learned my addr 172.21.15.161:0/771134566 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e1d9b640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8e4159ee0 0x7ff8e40caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.417 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e259c640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8cc004d00 0x7ff8e40b97c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e2d9d640 1 -- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8cc004d00 msgr2=0x7ff8e40b97c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e2d9d640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8cc004d00 0x7ff8e40b97c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e2d9d640 1 -- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8e4159ee0 msgr2=0x7ff8e40caf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e2d9d640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8e4159ee0 0x7ff8e40caf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e2d9d640 1 -- 172.21.15.161:0/771134566 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e40b3dc0 con 0x7ff8e41601b0 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e2d9d640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8e41601b0 0x7ff8e4160630 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc007360 tx=0x7ff8dc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8d2ffd640 1 -- 172.21.15.161:0/771134566 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc03d400 con 0x7ff8e41601b0 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8d2ffd640 1 -- 172.21.15.161:0/771134566 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff8dc03d580 con 0x7ff8e41601b0 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e40cb620 con 0x7ff8e41601b0 2024-03-23T14:37:29.418 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8d2ffd640 1 -- 172.21.15.161:0/771134566 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8dc03d930 con 0x7ff8e41601b0 2024-03-23T14:37:29.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.415+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e40cbbe0 con 0x7ff8e41601b0 2024-03-23T14:37:29.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.416+0000 7ff8d2ffd640 1 -- 172.21.15.161:0/771134566 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7ff8dc03b040 con 0x7ff8e41601b0 2024-03-23T14:37:29.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.417+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e40a1e00 con 0x7ff8e41601b0 2024-03-23T14:37:29.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.417+0000 7ff8d2ffd640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff8c406b810 0x7ff8c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.417+0000 7ff8d2ffd640 1 -- 172.21.15.161:0/771134566 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7ff8dc0ab3e0 con 0x7ff8e41601b0 2024-03-23T14:37:29.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.417+0000 7ff8e259c640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff8c406b810 0x7ff8c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.418+0000 7ff8e259c640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff8c406b810 0x7ff8c406dcd0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40011e0 tx=0x7ff8d4001170 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:29.419 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.421+0000 7ff8d2ffd640 1 -- 172.21.15.161:0/771134566 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7ff8dc075f40 con 0x7ff8e41601b0 2024-03-23T14:37:29.618 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.623+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff8e404f550 con 0x7ff8e41601b0 2024-03-23T14:37:29.619 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.624+0000 7ff8d2ffd640 1 -- 172.21.15.161:0/771134566 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7ff8dc047090 con 0x7ff8e41601b0 2024-03-23T14:37:29.620 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:29.620 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:29.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff8c406b810 msgr2=0x7ff8c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff8c406b810 0x7ff8c406dcd0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff8d40011e0 tx=0x7ff8d4001170 comp rx=0 tx=0).stop 2024-03-23T14:37:29.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8e41601b0 msgr2=0x7ff8e4160630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8e41601b0 0x7ff8e4160630 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7ff8dc007360 tx=0x7ff8dc004ab0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.622 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 shutdown_connections 2024-03-23T14:37:29.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7ff8c406b810 0x7ff8c406dcd0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7ff8e41601b0 0x7ff8e4160630 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7ff8e4159ee0 0x7ff8e40caf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 --2- 172.21.15.161:0/771134566 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7ff8cc004d00 0x7ff8e40b97c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.623 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 >> 172.21.15.161:0/771134566 conn(0x7ff8e409d190 msgr2=0x7ff8e40b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:29.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 shutdown_connections 2024-03-23T14:37:29.624 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.626+0000 7ff8e893e640 1 -- 172.21.15.161:0/771134566 wait complete. 2024-03-23T14:37:29.633 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:56.747667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:57.478793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:37:29.633 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:37:29.738 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.743+0000 7f82f7a23640 1 Processor -- start 2024-03-23T14:37:29.739 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.743+0000 7f82f7a23640 1 -- start start 2024-03-23T14:37:29.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.743+0000 7f82f7a23640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00ccca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.743+0000 7f82f7a23640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f82f00cc540 0x7f82f00cd1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.743+0000 7f82f7a23640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82f00cfd10 con 0x7f82f00ca8e0 2024-03-23T14:37:29.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f7a23640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f82f00cf4b0 con 0x7f82f00cc540 2024-03-23T14:37:29.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f7a23640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f82f002c670 con 0x7f82f00b1720 2024-03-23T14:37:29.740 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f57bf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00ccca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f4fbe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f82f00cc540 0x7f82f00cd1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f57bf640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00ccca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:58784/0 (socket says 172.21.15.161:58784) 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f4fbe640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f82f00cc540 0x7f82f00cd1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:39064/0 (socket says 172.21.15.161:39064) 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f57bf640 1 -- 172.21.15.161:0/151566304 learned_addr learned my addr 172.21.15.161:0/151566304 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f5fc0640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f82f00ca8e0 0x7f82f00cad70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:46152/0 (socket says 172.21.15.161:46152) 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f4fbe640 1 -- 172.21.15.161:0/151566304 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 msgr2=0x7f82f00ccca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f4fbe640 1 --2- 172.21.15.161:0/151566304 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00ccca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f4fbe640 1 -- 172.21.15.161:0/151566304 >> v1:172.21.15.183:6789/0 conn(0x7f82f00ca8e0 legacy=0x7f82f00cad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.741 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.744+0000 7f82f4fbe640 1 -- 172.21.15.161:0/151566304 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82f00715a0 con 0x7f82f00cc540 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.745+0000 7f82f4fbe640 1 --2- 172.21.15.161:0/151566304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00cc540 0x7f82f00cd1e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f82e40097a0 tx=0x7f82e402b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afce526265cef97f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.745+0000 7f82effff640 1 -- 172.21.15.161:0/151566304 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4031830 con 0x7f82f00cc540 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.745+0000 7f82effff640 1 -- 172.21.15.161:0/151566304 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e4031990 con 0x7f82f00cc540 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.745+0000 7f82effff640 1 -- 172.21.15.161:0/151566304 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e4031d20 con 0x7f82f00cc540 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.746+0000 7f82f7a23640 1 -- 172.21.15.161:0/151566304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00cc540 msgr2=0x7f82f00cd1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.746+0000 7f82f7a23640 1 --2- 172.21.15.161:0/151566304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00cc540 0x7f82f00cd1e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f82e40097a0 tx=0x7f82e402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.746+0000 7f82f7a23640 1 -- 172.21.15.161:0/151566304 shutdown_connections 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.746+0000 7f82f7a23640 1 --2- 172.21.15.161:0/151566304 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00cc540 0x7f82f00cd1e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.743 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.746+0000 7f82f7a23640 1 --2- 172.21.15.161:0/151566304 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00ccca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.746+0000 7f82f7a23640 1 -- 172.21.15.161:0/151566304 >> 172.21.15.161:0/151566304 conn(0x7f82f00b1e60 msgr2=0x7f82f00b2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:29.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.746+0000 7f82f7a23640 1 -- 172.21.15.161:0/151566304 shutdown_connections 2024-03-23T14:37:29.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.746+0000 7f82f7a23640 1 -- 172.21.15.161:0/151566304 wait complete. 2024-03-23T14:37:29.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.747+0000 7f82f7a23640 1 Processor -- start 2024-03-23T14:37:29.744 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.747+0000 7f82f7a23640 1 -- start start 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f7a23640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00c95f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f7a23640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00ca8e0 0x7f82f01f3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f7a23640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f00cc540 0x7f82f00cb680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f7a23640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f82f00c9ef0 con 0x7f82f00cc540 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f7a23640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f82f01f7080 con 0x7f82f00ca8e0 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f7a23640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f82f01f7360 con 0x7f82f00b1720 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f00cc540 0x7f82f00cb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f00cc540 0x7f82f00cb680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41844/0 (socket says 172.21.15.161:41844) 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 -- 172.21.15.161:0/932434020 learned_addr learned my addr 172.21.15.161:0/932434020 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f57bf640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00c95f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f4fbe640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00ca8e0 0x7f82f01f3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 -- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 msgr2=0x7f82f00c95f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00c95f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 -- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00ca8e0 msgr2=0x7f82f01f3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00ca8e0 0x7f82f01f3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 -- 172.21.15.161:0/932434020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82e4009400 con 0x7f82f00cc540 2024-03-23T14:37:29.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f4fbe640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00ca8e0 0x7f82f01f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:29.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.748+0000 7f82f5fc0640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f00cc540 0x7f82f00cb680 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c400 tx=0x7f82e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:29.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.749+0000 7f82edffb640 1 -- 172.21.15.161:0/932434020 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e00074e0 con 0x7f82f00cc540 2024-03-23T14:37:29.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.749+0000 7f82edffb640 1 -- 172.21.15.161:0/932434020 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f82e0007640 con 0x7f82f00cc540 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.749+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82f01f7640 con 0x7f82f00cc540 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.749+0000 7f82edffb640 1 -- 172.21.15.161:0/932434020 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82e00049e0 con 0x7f82f00cc540 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.749+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82f00c9b30 con 0x7f82f00cc540 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.750+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82f01f7830 con 0x7f82f00cc540 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.750+0000 7f82edffb640 1 -- 172.21.15.161:0/932434020 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f82e0004b40 con 0x7f82f00cc540 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.751+0000 7f82edffb640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f82dc06b810 0x7f82dc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.751+0000 7f82edffb640 1 -- 172.21.15.161:0/932434020 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f82e0089b40 con 0x7f82f00cc540 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.751+0000 7f82f57bf640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f82dc06b810 0x7f82dc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:29.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.752+0000 7f82f57bf640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f82dc06b810 0x7f82dc06dcd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009ce0 tx=0x7f82e800b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:29.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.755+0000 7f82edffb640 1 -- 172.21.15.161:0/932434020 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f82e00545f0 con 0x7f82f00cc540 2024-03-23T14:37:29.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.945+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f82f004f4e0 con 0x7f82f00cc540 2024-03-23T14:37:29.941 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.945+0000 7f82edffb640 1 -- 172.21.15.161:0/932434020 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7f82e0057cb0 con 0x7f82f00cc540 2024-03-23T14:37:29.941 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:37:29.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f82dc06b810 msgr2=0x7f82dc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f82dc06b810 0x7f82dc06dcd0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f82e8009ce0 tx=0x7f82e800b040 comp rx=0 tx=0).stop 2024-03-23T14:37:29.943 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f00cc540 msgr2=0x7f82f00cb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f00cc540 0x7f82f00cb680 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f82e000c400 tx=0x7f82e000c8c0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 shutdown_connections 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f82dc06b810 0x7f82dc06dcd0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f82f00cc540 0x7f82f00cb680 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f82f00ca8e0 0x7f82f01f3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 --2- 172.21.15.161:0/932434020 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f82f00b1720 0x7f82f00c95f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 >> 172.21.15.161:0/932434020 conn(0x7f82f00b1e60 msgr2=0x7f82f015f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 shutdown_connections 2024-03-23T14:37:29.944 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:29.947+0000 7f82f7a23640 1 -- 172.21.15.161:0/932434020 wait complete. 2024-03-23T14:37:29.954 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:37:30.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.063+0000 7f8df7d2c640 1 Processor -- start 2024-03-23T14:37:30.059 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.063+0000 7f8df7d2c640 1 -- start start 2024-03-23T14:37:30.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df7d2c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8df0161750 0x7f8df00d25c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df7d2c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00d1330 con 0x7f8df00d1cb0 2024-03-23T14:37:30.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df7d2c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df0162e80 con 0x7f8df0161b30 2024-03-23T14:37:30.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df7d2c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f8df00d0720 con 0x7f8df0161750 2024-03-23T14:37:30.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df5ac8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f8df0161750 0x7f8df00d25c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df52c7640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8df00d1cb0 0x7f8df0162060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35506/0 (socket says 172.21.15.161:35506) 2024-03-23T14:37:30.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df52c7640 1 -- 172.21.15.161:0/1797413705 learned_addr learned my addr 172.21.15.161:0/1797413705 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:30.060 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df5ac8640 1 -- 172.21.15.161:0/1797413705 >> v1:172.21.15.183:6789/0 conn(0x7f8df0161b30 legacy=0x7f8df0162770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df5ac8640 1 -- 172.21.15.161:0/1797413705 >> v1:172.21.15.161:6789/0 conn(0x7f8df00d1cb0 legacy=0x7f8df0162060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df4ac6640 1 -- 172.21.15.161:0/1797413705 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3625094124 0 0) 0x7f8df00d1330 con 0x7f8df00d1cb0 2024-03-23T14:37:30.061 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.064+0000 7f8df5ac8640 1 -- 172.21.15.161:0/1797413705 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df00d0a00 con 0x7f8df0161750 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.065+0000 7f8df5ac8640 1 --2- 172.21.15.161:0/1797413705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161750 0x7f8df00d25c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8de00097a0 tx=0x7f8de002b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604ac53698899045 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.065+0000 7f8df4ac6640 1 -- 172.21.15.161:0/1797413705 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0030610 con 0x7f8df0161750 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.065+0000 7f8df4ac6640 1 -- 172.21.15.161:0/1797413705 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de0030770 con 0x7f8df0161750 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.065+0000 7f8df4ac6640 1 -- 172.21.15.161:0/1797413705 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de0030b00 con 0x7f8df0161750 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.066+0000 7f8df7d2c640 1 -- 172.21.15.161:0/1797413705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161750 msgr2=0x7f8df00d25c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.066+0000 7f8df7d2c640 1 --2- 172.21.15.161:0/1797413705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161750 0x7f8df00d25c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7f8de00097a0 tx=0x7f8de002b960 comp rx=0 tx=0).stop 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.066+0000 7f8df7d2c640 1 -- 172.21.15.161:0/1797413705 shutdown_connections 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.066+0000 7f8df7d2c640 1 --2- 172.21.15.161:0/1797413705 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161750 0x7f8df00d25c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.066+0000 7f8df7d2c640 1 -- 172.21.15.161:0/1797413705 >> 172.21.15.161:0/1797413705 conn(0x7f8df009d450 msgr2=0x7f8df00d40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.066+0000 7f8df7d2c640 1 -- 172.21.15.161:0/1797413705 shutdown_connections 2024-03-23T14:37:30.063 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.066+0000 7f8df7d2c640 1 -- 172.21.15.161:0/1797413705 wait complete. 2024-03-23T14:37:30.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.067+0000 7f8df7d2c640 1 Processor -- start 2024-03-23T14:37:30.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.067+0000 7f8df7d2c640 1 -- start start 2024-03-23T14:37:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df7d2c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8df00d1cb0 0x7f8df00cdbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df7d2c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8df0161750 0x7f8df00c9d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df7d2c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161b30 0x7f8df0151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df7d2c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f8df00ce110 con 0x7f8df0161b30 2024-03-23T14:37:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df7d2c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f8df00cbc60 con 0x7f8df0161750 2024-03-23T14:37:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df7d2c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f8df00cbf00 con 0x7f8df00d1cb0 2024-03-23T14:37:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df62c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161b30 0x7f8df0151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.066 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df62c9640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161b30 0x7f8df0151f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41858/0 (socket says 172.21.15.161:41858) 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df62c9640 1 -- 172.21.15.161:0/371692405 learned_addr learned my addr 172.21.15.161:0/371692405 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df5ac8640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8df00d1cb0 0x7f8df00cdbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df52c7640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8df0161750 0x7f8df00c9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df62c9640 1 -- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8df00d1cb0 msgr2=0x7f8df00cdbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df62c9640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8df00d1cb0 0x7f8df00cdbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df62c9640 1 -- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8df0161750 msgr2=0x7f8df00c9d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df62c9640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8df0161750 0x7f8df00c9d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.068+0000 7f8df62c9640 1 -- 172.21.15.161:0/371692405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8de0009400 con 0x7f8df0161b30 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.069+0000 7f8df62c9640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161b30 0x7f8df0151f10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8de800c680 tx=0x7f8de800cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.069+0000 7f8de67fc640 1 -- 172.21.15.161:0/371692405 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8003b70 con 0x7f8df0161b30 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.069+0000 7f8de67fc640 1 -- 172.21.15.161:0/371692405 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8de8003cd0 con 0x7f8df0161b30 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.069+0000 7f8de67fc640 1 -- 172.21.15.161:0/371692405 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8de8005310 con 0x7f8df0161b30 2024-03-23T14:37:30.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.069+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8df00cc1e0 con 0x7f8df0161b30 2024-03-23T14:37:30.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.069+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8df00ce340 con 0x7f8df0161b30 2024-03-23T14:37:30.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.070+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8df00a2030 con 0x7f8df0161b30 2024-03-23T14:37:30.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.070+0000 7f8de67fc640 1 -- 172.21.15.161:0/371692405 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f8de8005470 con 0x7f8df0161b30 2024-03-23T14:37:30.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.071+0000 7f8de67fc640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.071+0000 7f8de67fc640 1 -- 172.21.15.161:0/371692405 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f8de8089df0 con 0x7f8df0161b30 2024-03-23T14:37:30.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.071+0000 7f8df5ac8640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.069 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.071+0000 7f8df5ac8640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8de0036000 tx=0x7f8de0008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.070 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.075+0000 7f8de67fc640 1 -- 172.21.15.161:0/371692405 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f8de80548c0 con 0x7f8df0161b30 2024-03-23T14:37:30.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.265+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8df004f550 con 0x7f8df0161b30 2024-03-23T14:37:30.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.265+0000 7f8de67fc640 1 -- 172.21.15.161:0/371692405 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f8de8057f80 con 0x7f8df0161b30 2024-03-23T14:37:30.261 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:37:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 msgr2=0x7f8dd406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f8de0036000 tx=0x7f8de0008030 comp rx=0 tx=0).stop 2024-03-23T14:37:30.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161b30 msgr2=0x7f8df0151f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161b30 0x7f8df0151f10 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7f8de800c680 tx=0x7f8de800cb40 comp rx=0 tx=0).stop 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 shutdown_connections 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f8dd406b810 0x7f8dd406dcd0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f8df0161b30 0x7f8df0151f10 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8df0161750 0x7f8df00c9d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 --2- 172.21.15.161:0/371692405 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f8df00d1cb0 0x7f8df00cdbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 >> 172.21.15.161:0/371692405 conn(0x7f8df009d450 msgr2=0x7f8df015fae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 shutdown_connections 2024-03-23T14:37:30.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.267+0000 7f8df7d2c640 1 -- 172.21.15.161:0/371692405 wait complete. 2024-03-23T14:37:30.274 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:37:30.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.387+0000 7f229fa57640 1 Processor -- start 2024-03-23T14:37:30.383 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.387+0000 7f229fa57640 1 -- start start 2024-03-23T14:37:30.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229fa57640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22980d1270 0x7f22980d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229fa57640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2298161c00 con 0x7f22980d29d0 2024-03-23T14:37:30.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229fa57640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22980d2e20 con 0x7f2298161820 2024-03-23T14:37:30.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229fa57640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f22980cea30 con 0x7f22980d1270 2024-03-23T14:37:30.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229cff2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22980d1270 0x7f22980d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229dff4640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f22980d29d0 0x7f22980b0a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:35512/0 (socket says 172.21.15.161:35512) 2024-03-23T14:37:30.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229cff2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f22980d1270 0x7f22980d2490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41864/0 (socket says 172.21.15.161:41864) 2024-03-23T14:37:30.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229dff4640 1 -- 172.21.15.161:0/3947961645 learned_addr learned my addr 172.21.15.161:0/3947961645 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:30.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229cff2640 1 -- 172.21.15.161:0/3947961645 >> v1:172.21.15.183:6789/0 conn(0x7f2298161820 legacy=0x7f22980d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:30.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229cff2640 1 -- 172.21.15.161:0/3947961645 >> v1:172.21.15.161:6789/0 conn(0x7f22980d29d0 legacy=0x7f22980b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.388+0000 7f229cff2640 1 -- 172.21.15.161:0/3947961645 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22980cfd70 con 0x7f22980d1270 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.389+0000 7f229cff2640 1 --2- 172.21.15.161:0/3947961645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f22980d2490 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f22880097a0 tx=0x7f228802b710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad6fc3ba5379c281 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.390+0000 7f228ffff640 1 -- 172.21.15.161:0/3947961645 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288030710 con 0x7f22980d1270 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.390+0000 7f228ffff640 1 -- 172.21.15.161:0/3947961645 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f228802e050 con 0x7f22980d1270 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.390+0000 7f228ffff640 1 -- 172.21.15.161:0/3947961645 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2288030a90 con 0x7f22980d1270 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.390+0000 7f229fa57640 1 -- 172.21.15.161:0/3947961645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 msgr2=0x7f22980d2490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.390+0000 7f229fa57640 1 --2- 172.21.15.161:0/3947961645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f22980d2490 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f22880097a0 tx=0x7f228802b710 comp rx=0 tx=0).stop 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.390+0000 7f229fa57640 1 -- 172.21.15.161:0/3947961645 shutdown_connections 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.390+0000 7f229fa57640 1 --2- 172.21.15.161:0/3947961645 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f22980d2490 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.390+0000 7f229fa57640 1 -- 172.21.15.161:0/3947961645 >> 172.21.15.161:0/3947961645 conn(0x7f229809d190 msgr2=0x7f22980d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:30.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.391+0000 7f229fa57640 1 -- 172.21.15.161:0/3947961645 shutdown_connections 2024-03-23T14:37:30.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.391+0000 7f229fa57640 1 -- 172.21.15.161:0/3947961645 wait complete. 2024-03-23T14:37:30.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.392+0000 7f229fa57640 1 Processor -- start 2024-03-23T14:37:30.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.392+0000 7f229fa57640 1 -- start start 2024-03-23T14:37:30.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229fa57640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f229815c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229fa57640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22980d29d0 0x7f22980b40f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229fa57640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2298161820 0x7f22980cb090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229fa57640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f22980c83f0 con 0x7f22980d1270 2024-03-23T14:37:30.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229fa57640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f22980c86d0 con 0x7f2298161820 2024-03-23T14:37:30.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229fa57640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f22980c89b0 con 0x7f22980d29d0 2024-03-23T14:37:30.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229d7f3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f229815c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.391 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229d7f3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f229815c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:41874/0 (socket says 172.21.15.161:41874) 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229d7f3640 1 -- 172.21.15.161:0/3161525102 learned_addr learned my addr 172.21.15.161:0/3161525102 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229cff2640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22980d29d0 0x7f22980b40f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.393+0000 7f229dff4640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2298161820 0x7f22980cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229d7f3640 1 -- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22980d29d0 msgr2=0x7f22980b40f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229d7f3640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22980d29d0 0x7f22980b40f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229d7f3640 1 -- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2298161820 msgr2=0x7f22980cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229d7f3640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2298161820 0x7f22980cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229d7f3640 1 -- 172.21.15.161:0/3161525102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2288009400 con 0x7f22980d1270 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229cff2640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22980d29d0 0x7f22980b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229dff4640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2298161820 0x7f22980cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229d7f3640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f229815c1a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f229400b3f0 tx=0x7f229400b8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f228dffb640 1 -- 172.21.15.161:0/3161525102 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294011830 con 0x7f22980d1270 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f228dffb640 1 -- 172.21.15.161:0/3161525102 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2294011990 con 0x7f22980d1270 2024-03-23T14:37:30.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22980c8c90 con 0x7f22980d1270 2024-03-23T14:37:30.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f228dffb640 1 -- 172.21.15.161:0/3161525102 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2294011d40 con 0x7f22980d1270 2024-03-23T14:37:30.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.394+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f229815c960 con 0x7f22980d1270 2024-03-23T14:37:30.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.395+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f229815cb50 con 0x7f22980d1270 2024-03-23T14:37:30.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.396+0000 7f228dffb640 1 -- 172.21.15.161:0/3161525102 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f229401a400 con 0x7f22980d1270 2024-03-23T14:37:30.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.396+0000 7f228dffb640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f227c06b810 0x7f227c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.396+0000 7f228dffb640 1 -- 172.21.15.161:0/3161525102 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f229408a090 con 0x7f22980d1270 2024-03-23T14:37:30.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.397+0000 7f229cff2640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f227c06b810 0x7f227c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.397+0000 7f229cff2640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f227c06b810 0x7f227c06dcd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2288034000 tx=0x7f2288008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.401+0000 7f228dffb640 1 -- 172.21.15.161:0/3161525102 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f2294054a20 con 0x7f22980d1270 2024-03-23T14:37:30.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.579+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f229804f4e0 con 0x7f22980d1270 2024-03-23T14:37:30.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.579+0000 7f228dffb640 1 -- 172.21.15.161:0/3161525102 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7f22940150a0 con 0x7f22980d1270 2024-03-23T14:37:30.575 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:37:30.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f227c06b810 msgr2=0x7f227c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f227c06b810 0x7f227c06dcd0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2288034000 tx=0x7f2288008030 comp rx=0 tx=0).stop 2024-03-23T14:37:30.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 msgr2=0x7f229815c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.577 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f229815c1a0 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f229400b3f0 tx=0x7f229400b8b0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 shutdown_connections 2024-03-23T14:37:30.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f227c06b810 0x7f227c06dcd0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2298161820 0x7f22980cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f22980d29d0 0x7f22980b40f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 --2- 172.21.15.161:0/3161525102 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f22980d1270 0x7f229815c1a0 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 >> 172.21.15.161:0/3161525102 conn(0x7f229809d190 msgr2=0x7f22980d47e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:30.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 shutdown_connections 2024-03-23T14:37:30.578 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.581+0000 7f229fa57640 1 -- 172.21.15.161:0/3161525102 wait complete. 2024-03-23T14:37:30.588 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-23T14:37:30.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.694+0000 7fcc5066f640 1 Processor -- start 2024-03-23T14:37:30.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.694+0000 7fcc5066f640 1 -- start start 2024-03-23T14:37:30.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc5066f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcc4815e400 0x7fcc480cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc5066f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fcc480cf9d0 0x7fcc480d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc5066f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc480d1470 0x7fcc480d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc5066f640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fcc480d1050 con 0x7fcc4815e400 2024-03-23T14:37:30.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc5066f640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fcc480d0460 con 0x7fcc480cf9d0 2024-03-23T14:37:30.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc5066f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fcc480d06b0 con 0x7fcc480d1470 2024-03-23T14:37:30.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc4e40b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcc4815e400 0x7fcc480cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc4e40b640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fcc4815e400 0x7fcc480cf490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60452/0 (socket says 172.21.15.161:60452) 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc4e40b640 1 -- 172.21.15.161:0/174669694 learned_addr learned my addr 172.21.15.161:0/174669694 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc4ec0c640 1 --2- 172.21.15.161:0/174669694 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc480d1470 0x7fcc480d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.695+0000 7fcc4dc0a640 1 --2- 172.21.15.161:0/174669694 >> v2:172.21.15.183:3300/0 conn(0x7fcc480cf9d0 0x7fcc480d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.696+0000 7fcc4e40b640 1 -- 172.21.15.161:0/174669694 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc480d1470 msgr2=0x7fcc480d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.696+0000 7fcc4e40b640 1 --2- 172.21.15.161:0/174669694 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc480d1470 0x7fcc480d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.696+0000 7fcc4e40b640 1 -- 172.21.15.161:0/174669694 >> v2:172.21.15.183:3300/0 conn(0x7fcc480cf9d0 msgr2=0x7fcc480d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.696+0000 7fcc4e40b640 1 --2- 172.21.15.161:0/174669694 >> v2:172.21.15.183:3300/0 conn(0x7fcc480cf9d0 0x7fcc480d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.696+0000 7fcc4e40b640 1 -- 172.21.15.161:0/174669694 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc480d0990 con 0x7fcc4815e400 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.696+0000 7fcc4ec0c640 1 --2- 172.21.15.161:0/174669694 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc480d1470 0x7fcc480d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.696+0000 7fcc4dc0a640 1 --2- 172.21.15.161:0/174669694 >> v2:172.21.15.183:3300/0 conn(0x7fcc480cf9d0 0x7fcc480d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:30.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.696+0000 7fcc4e40b640 1 --2- 172.21.15.161:0/174669694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc4815e400 0x7fcc480cf490 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fcc480b6a20 tx=0x7fcc3802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=467ede1fcc57a283 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc4d409640 1 -- 172.21.15.161:0/174669694 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc38031870 con 0x7fcc4815e400 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc4d409640 1 -- 172.21.15.161:0/174669694 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc380319d0 con 0x7fcc4815e400 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc4d409640 1 -- 172.21.15.161:0/174669694 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc38031d60 con 0x7fcc4815e400 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 -- 172.21.15.161:0/174669694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc4815e400 msgr2=0x7fcc480cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 --2- 172.21.15.161:0/174669694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc4815e400 0x7fcc480cf490 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7fcc480b6a20 tx=0x7fcc3802b910 comp rx=0 tx=0).stop 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 -- 172.21.15.161:0/174669694 shutdown_connections 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 --2- 172.21.15.161:0/174669694 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc480d1470 0x7fcc480d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 --2- 172.21.15.161:0/174669694 >> v2:172.21.15.183:3300/0 conn(0x7fcc480cf9d0 0x7fcc480d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 --2- 172.21.15.161:0/174669694 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc4815e400 0x7fcc480cf490 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 -- 172.21.15.161:0/174669694 >> 172.21.15.161:0/174669694 conn(0x7fcc4809d190 msgr2=0x7fcc480d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 -- 172.21.15.161:0/174669694 shutdown_connections 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.697+0000 7fcc5066f640 1 -- 172.21.15.161:0/174669694 wait complete. 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.698+0000 7fcc5066f640 1 Processor -- start 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.698+0000 7fcc5066f640 1 -- start start 2024-03-23T14:37:30.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc5066f640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc480cf9d0 0x7fcc481594d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc5066f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc480d1470 0x7fcc48160d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc5066f640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc4815e400 0x7fcc4815b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc5066f640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fcc480c8910 con 0x7fcc480d1470 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc5066f640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fcc480c8bf0 con 0x7fcc480cf9d0 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc5066f640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fcc480c8ed0 con 0x7fcc4815e400 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc4dc0a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc480d1470 0x7fcc48160d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc4dc0a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc480d1470 0x7fcc48160d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60462/0 (socket says 172.21.15.161:60462) 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc4dc0a640 1 -- 172.21.15.161:0/3558321435 learned_addr learned my addr 172.21.15.161:0/3558321435 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc4e40b640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc480cf9d0 0x7fcc481594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.699+0000 7fcc4ec0c640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc4815e400 0x7fcc4815b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc4dc0a640 1 -- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc4815e400 msgr2=0x7fcc4815b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc4dc0a640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc4815e400 0x7fcc4815b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc4dc0a640 1 -- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc480cf9d0 msgr2=0x7fcc481594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc4dc0a640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc480cf9d0 0x7fcc481594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc4dc0a640 1 -- 172.21.15.161:0/3558321435 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc380094d0 con 0x7fcc480d1470 2024-03-23T14:37:30.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc4dc0a640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc480d1470 0x7fcc48160d40 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400e810 tx=0x7fcc4400ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc3effd640 1 -- 172.21.15.161:0/3558321435 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc44004030 con 0x7fcc480d1470 2024-03-23T14:37:30.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc3effd640 1 -- 172.21.15.161:0/3558321435 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcc44004190 con 0x7fcc480d1470 2024-03-23T14:37:30.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc480c91b0 con 0x7fcc480d1470 2024-03-23T14:37:30.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc3effd640 1 -- 172.21.15.161:0/3558321435 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc44004520 con 0x7fcc480d1470 2024-03-23T14:37:30.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.700+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc48159a10 con 0x7fcc480d1470 2024-03-23T14:37:30.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.701+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc4806b9a0 con 0x7fcc480d1470 2024-03-23T14:37:30.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.701+0000 7fcc3effd640 1 -- 172.21.15.161:0/3558321435 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fcc4400c6d0 con 0x7fcc480d1470 2024-03-23T14:37:30.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.702+0000 7fcc3effd640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcc2c06b810 0x7fcc2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:30.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.702+0000 7fcc3effd640 1 -- 172.21.15.161:0/3558321435 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fcc44089cc0 con 0x7fcc480d1470 2024-03-23T14:37:30.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.702+0000 7fcc4e40b640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcc2c06b810 0x7fcc2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:30.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.703+0000 7fcc4e40b640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcc2c06b810 0x7fcc2c06dcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcc38036000 tx=0x7fcc3803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:30.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.706+0000 7fcc3effd640 1 -- 172.21.15.161:0/3558321435 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fcc44054770 con 0x7fcc480d1470 2024-03-23T14:37:30.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.896+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fcc480a1e00 con 0x7fcc480d1470 2024-03-23T14:37:30.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.896+0000 7fcc3effd640 1 -- 172.21.15.161:0/3558321435 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v48) ==== 80+0+0 (secure 0 0 0) 0x7fcc44020080 con 0x7fcc480d1470 2024-03-23T14:37:30.892 INFO:teuthology.orchestra.run.smithi161.stderr:listed 0 entries 2024-03-23T14:37:30.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcc2c06b810 msgr2=0x7fcc2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcc2c06b810 0x7fcc2c06dcd0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7fcc38036000 tx=0x7fcc3803c040 comp rx=0 tx=0).stop 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc480d1470 msgr2=0x7fcc48160d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc480d1470 0x7fcc48160d40 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7fcc4400e810 tx=0x7fcc4400ecd0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 shutdown_connections 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fcc2c06b810 0x7fcc2c06dcd0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fcc4815e400 0x7fcc4815b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fcc480d1470 0x7fcc48160d40 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.898+0000 7fcc5066f640 1 --2- 172.21.15.161:0/3558321435 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fcc480cf9d0 0x7fcc481594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.899+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 >> 172.21.15.161:0/3558321435 conn(0x7fcc4809d190 msgr2=0x7fcc4815c370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.899+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 shutdown_connections 2024-03-23T14:37:30.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:30.899+0000 7fcc5066f640 1 -- 172.21.15.161:0/3558321435 wait complete. 2024-03-23T14:37:30.905 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi161.front.sepia.ceph.com... 2024-03-23T14:37:30.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:30.905 DEBUG:teuthology.orchestra.run.smithi161:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-03-23T14:37:30.930 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi161.stderr:ceph-fuse[58938]: fuse finished with error 0 and tester_r 0 2024-03-23T14:37:30.932 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:37:30.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:30.933 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-23T14:37:31.100 INFO:tasks.cephfs.mount:Terminating background process 2024-03-23T14:37:31.100 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:37:31.100 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:31.119 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056359,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17751 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T14:37:31.119 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:37:31.119 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:31.120 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2024-03-23T14:37:31.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:37:31.178 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2024-03-23T14:37:31.178 INFO:tasks.cephfs.mount:Terminating background process 2024-03-23T14:37:31.178 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:37:31.178 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:31.193 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18767 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T14:37:31.193 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:37:31.193 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:31.193 DEBUG:teuthology.orchestra.run.smithi161:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2024-03-23T14:37:31.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:37:31.248 INFO:teuthology.orchestra.run.smithi161.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2024-03-23T14:37:31.249 INFO:tasks.cephfs.mount:Terminating background process 2024-03-23T14:37:31.249 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:37:31.249 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:31.267 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056359,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17751 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T14:37:31.267 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:37:31.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:31.267 DEBUG:teuthology.orchestra.run.smithi183:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.3) 2024-03-23T14:37:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:37:31.324 INFO:teuthology.orchestra.run.smithi183.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.3': No such file or directory 2024-03-23T14:37:31.325 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:37:31.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.431+0000 7f1ea5ca4640 1 Processor -- start 2024-03-23T14:37:31.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.431+0000 7f1ea5ca4640 1 -- start start 2024-03-23T14:37:31.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1ea5ca4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ea00d1470 0x7f1ea00d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1ea5ca4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f1ea00d2970 0x7f1ea00b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1ea5ca4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ea00d0460 con 0x7f1ea015e400 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1ea5ca4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea00d1050 con 0x7f1ea00d1470 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1ea5ca4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f1ea00d2e00 con 0x7f1ea00d2970 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1e9effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ea00d1470 0x7f1ea00d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1e9effd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f1ea00d1470 0x7f1ea00d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60466/0 (socket says 172.21.15.161:60466) 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1e9effd640 1 -- 172.21.15.161:0/1688312312 learned_addr learned my addr 172.21.15.161:0/1688312312 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1e9ffff640 1 --2- 172.21.15.161:0/1688312312 >> v2:172.21.15.183:3300/0 conn(0x7f1ea00d2970 0x7f1ea00b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1e9effd640 1 -- 172.21.15.161:0/1688312312 >> v2:172.21.15.183:3300/0 conn(0x7f1ea00d2970 msgr2=0x7f1ea00b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1e9effd640 1 --2- 172.21.15.161:0/1688312312 >> v2:172.21.15.183:3300/0 conn(0x7f1ea00d2970 0x7f1ea00b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1e9effd640 1 -- 172.21.15.161:0/1688312312 >> v1:172.21.15.183:6789/0 conn(0x7f1ea015e400 legacy=0x7f1ea00cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.432+0000 7f1e9effd640 1 -- 172.21.15.161:0/1688312312 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea00d06b0 con 0x7f1ea00d1470 2024-03-23T14:37:31.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.433+0000 7f1e9effd640 1 --2- 172.21.15.161:0/1688312312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d1470 0x7f1ea00d2430 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00d00e0 tx=0x7f1e9002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ac12a12b45f5698 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1e9e7fc640 1 -- 172.21.15.161:0/1688312312 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90030860 con 0x7f1ea00d1470 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1e9e7fc640 1 -- 172.21.15.161:0/1688312312 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e900309c0 con 0x7f1ea00d1470 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1e9e7fc640 1 -- 172.21.15.161:0/1688312312 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90030d50 con 0x7f1ea00d1470 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/1688312312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d1470 msgr2=0x7f1ea00d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/1688312312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d1470 0x7f1ea00d2430 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00d00e0 tx=0x7f1e9002b9a0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/1688312312 shutdown_connections 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/1688312312 >> v2:172.21.15.183:3300/0 conn(0x7f1ea00d2970 0x7f1ea00b0a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/1688312312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d1470 0x7f1ea00d2430 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/1688312312 >> 172.21.15.161:0/1688312312 conn(0x7f1ea009d190 msgr2=0x7f1ea00d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/1688312312 shutdown_connections 2024-03-23T14:37:31.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.434+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/1688312312 wait complete. 2024-03-23T14:37:31.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.436+0000 7f1ea5ca4640 1 Processor -- start 2024-03-23T14:37:31.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.436+0000 7f1ea5ca4640 1 -- start start 2024-03-23T14:37:31.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.437+0000 7f1ea5ca4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea00d1470 0x7f1ea015cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.437+0000 7f1ea5ca4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d2970 0x7f1ea01f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.437+0000 7f1ea5ca4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea015e400 0x7f1ea00c8220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.437+0000 7f1ea5ca4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea00ce310 con 0x7f1ea00d2970 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.437+0000 7f1ea5ca4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea015d240 con 0x7f1ea015e400 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.437+0000 7f1ea5ca4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f1ea01f45f0 con 0x7f1ea00d1470 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.437+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d2970 0x7f1ea01f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d2970 0x7f1ea01f3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60482/0 (socket says 172.21.15.161:60482) 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9effd640 1 -- 172.21.15.161:0/4120050988 learned_addr learned my addr 172.21.15.161:0/4120050988 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9ffff640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea015e400 0x7f1ea00c8220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9f7fe640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea00d1470 0x7f1ea015cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9effd640 1 -- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea00d1470 msgr2=0x7f1ea015cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9effd640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea00d1470 0x7f1ea015cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9effd640 1 -- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea015e400 msgr2=0x7f1ea00c8220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9effd640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea015e400 0x7f1ea00c8220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9effd640 1 -- 172.21.15.161:0/4120050988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e900094d0 con 0x7f1ea00d2970 2024-03-23T14:37:31.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e9effd640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d2970 0x7f1ea01f3c20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1e90000c00 tx=0x7f1e90004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:31.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e87fff640 1 -- 172.21.15.161:0/4120050988 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e90039a90 con 0x7f1ea00d2970 2024-03-23T14:37:31.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e87fff640 1 -- 172.21.15.161:0/4120050988 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1e90039bf0 con 0x7f1ea00d2970 2024-03-23T14:37:31.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea01f48d0 con 0x7f1ea00d2970 2024-03-23T14:37:31.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1e87fff640 1 -- 172.21.15.161:0/4120050988 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e900433f0 con 0x7f1ea00d2970 2024-03-23T14:37:31.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.438+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea00c7a20 con 0x7f1ea00d2970 2024-03-23T14:37:31.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.439+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea00a1e00 con 0x7f1ea00d2970 2024-03-23T14:37:31.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.440+0000 7f1e87fff640 1 -- 172.21.15.161:0/4120050988 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f1e9004c3b0 con 0x7f1ea00d2970 2024-03-23T14:37:31.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.440+0000 7f1e87fff640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e8c06b810 0x7f1e8c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.440+0000 7f1e87fff640 1 -- 172.21.15.161:0/4120050988 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f1e900ac5f0 con 0x7f1ea00d2970 2024-03-23T14:37:31.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.440+0000 7f1e9f7fe640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e8c06b810 0x7f1e8c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.437 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.441+0000 7f1e9f7fe640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e8c06b810 0x7f1e8c06dcd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1e94007fd0 tx=0x7f1e94009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:31.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.449+0000 7f1e87fff640 1 -- 172.21.15.161:0/4120050988 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f1e900770c0 con 0x7f1ea00d2970 2024-03-23T14:37:31.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.660+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1ea004f550 con 0x7f1ea00d2970 2024-03-23T14:37:31.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.660+0000 7f1e87fff640 1 -- 172.21.15.161:0/4120050988 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5193 (secure 0 0 0) 0x7f1e9007a780 con 0x7f1ea00d2970 2024-03-23T14:37:31.656 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:31.657 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2496234294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":2496234294},{"type":"v1","addr":"172.21.15.183:6836","nonce":2496234294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5396,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2645821131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2645821131},{"type":"v1","addr":"172.21.15.183:6837","nonce":2645821131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/2183619206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2183619206},{"type":"v1","addr":"172.21.15.161:6834","nonce":2183619206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:58.503230+0000","modified":"2024-03-23T14:37:00.601296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/1671955402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":1671955402},{"type":"v1","addr":"172.21.15.161:6835","nonce":1671955402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5925,"qdb_cluster":[5925]},"id":3}]} 2024-03-23T14:37:31.657 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-23T14:37:31.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.662+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e8c06b810 msgr2=0x7f1e8c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.662+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e8c06b810 0x7f1e8c06dcd0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1e94007fd0 tx=0x7f1e94009040 comp rx=0 tx=0).stop 2024-03-23T14:37:31.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d2970 msgr2=0x7f1ea01f3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d2970 0x7f1ea01f3c20 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f1e90000c00 tx=0x7f1e90004730 comp rx=0 tx=0).stop 2024-03-23T14:37:31.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 shutdown_connections 2024-03-23T14:37:31.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f1e8c06b810 0x7f1e8c06dcd0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f1ea015e400 0x7f1ea00c8220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.661 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f1ea00d2970 0x7f1ea01f3c20 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 --2- 172.21.15.161:0/4120050988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f1ea00d1470 0x7f1ea015cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 >> 172.21.15.161:0/4120050988 conn(0x7f1ea009d190 msgr2=0x7f1ea006d090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:31.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 shutdown_connections 2024-03-23T14:37:31.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.663+0000 7f1ea5ca4640 1 -- 172.21.15.161:0/4120050988 wait complete. 2024-03-23T14:37:31.670 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.775+0000 7fc3ad7f6640 1 Processor -- start 2024-03-23T14:37:31.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.775+0000 7fc3ad7f6640 1 -- start start 2024-03-23T14:37:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.775+0000 7fc3ad7f6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc3a815e400 0x7fc3a80cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3ad7f6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a80d0460 con 0x7fc3a80cf9d0 2024-03-23T14:37:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3ad7f6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3a80d06b0 con 0x7fc3a80d2b40 2024-03-23T14:37:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3ad7f6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc3a80d1050 con 0x7fc3a815e400 2024-03-23T14:37:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc39ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fc3a80cf9d0 0x7fc3a80d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42768/0 (socket says 172.21.15.161:42768) 2024-03-23T14:37:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc39ffff640 1 -- 172.21.15.161:0/2845119532 learned_addr learned my addr 172.21.15.161:0/2845119532 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:31.772 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3a6ffd640 1 --2- 172.21.15.161:0/2845119532 >> v2:172.21.15.183:3300/0 conn(0x7fc3a815e400 0x7fc3a80cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3a67fc640 1 -- 172.21.15.161:0/2845119532 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1770953146 0 0) 0x7fc3a80d0460 con 0x7fc3a80cf9d0 2024-03-23T14:37:31.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3a67fc640 1 -- 172.21.15.161:0/2845119532 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc394003520 con 0x7fc3a80cf9d0 2024-03-23T14:37:31.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3a6ffd640 1 -- 172.21.15.161:0/2845119532 >> v1:172.21.15.183:6789/0 conn(0x7fc3a80d2b40 legacy=0x7fc3a80b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3a67fc640 1 -- 172.21.15.161:0/2845119532 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2423374331 0 0) 0x7fc3a80d06b0 con 0x7fc3a80d2b40 2024-03-23T14:37:31.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3a6ffd640 1 -- 172.21.15.161:0/2845119532 >> v1:172.21.15.161:6789/0 conn(0x7fc3a80cf9d0 legacy=0x7fc3a80d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.776+0000 7fc3a6ffd640 1 -- 172.21.15.161:0/2845119532 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a80d0990 con 0x7fc3a815e400 2024-03-23T14:37:31.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.777+0000 7fc3a6ffd640 1 --2- 172.21.15.161:0/2845119532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a815e400 0x7fc3a80cf490 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc3980097a0 tx=0x7fc39802b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f38974d1bc9dd96 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:31.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.777+0000 7fc3a67fc640 1 -- 172.21.15.161:0/2845119532 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc398030610 con 0x7fc3a815e400 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.777+0000 7fc3a67fc640 1 -- 172.21.15.161:0/2845119532 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc398030770 con 0x7fc3a815e400 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.777+0000 7fc3a67fc640 1 -- 172.21.15.161:0/2845119532 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc398030ae0 con 0x7fc3a815e400 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.778+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2845119532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a815e400 msgr2=0x7fc3a80cf490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.778+0000 7fc3ad7f6640 1 --2- 172.21.15.161:0/2845119532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a815e400 0x7fc3a80cf490 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fc3980097a0 tx=0x7fc39802b960 comp rx=0 tx=0).stop 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.778+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2845119532 shutdown_connections 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.778+0000 7fc3ad7f6640 1 --2- 172.21.15.161:0/2845119532 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a815e400 0x7fc3a80cf490 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.778+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2845119532 >> 172.21.15.161:0/2845119532 conn(0x7fc3a809d190 msgr2=0x7fc3a80d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.778+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2845119532 shutdown_connections 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.778+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2845119532 wait complete. 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.779+0000 7fc3ad7f6640 1 Processor -- start 2024-03-23T14:37:31.775 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.779+0000 7fc3ad7f6640 1 -- start start 2024-03-23T14:37:31.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3ad7f6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a80cf9d0 0x7fc3a814f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3ad7f6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3a80d2b40 0x7fc3a815e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3ad7f6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3a815e400 0x7fc3a80b36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3ad7f6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a80ca100 con 0x7fc3a815e400 2024-03-23T14:37:31.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3ad7f6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a80c9080 con 0x7fc3a80cf9d0 2024-03-23T14:37:31.778 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3ad7f6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc3a80c9360 con 0x7fc3a80d2b40 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3a815e400 0x7fc3a80b36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3a815e400 0x7fc3a80b36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60498/0 (socket says 172.21.15.161:60498) 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 -- 172.21.15.161:0/2443763530 learned_addr learned my addr 172.21.15.161:0/2443763530 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a6ffd640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a80cf9d0 0x7fc3a814f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc39ffff640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3a80d2b40 0x7fc3a815e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 -- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3a80d2b40 msgr2=0x7fc3a815e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3a80d2b40 0x7fc3a815e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 -- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a80cf9d0 msgr2=0x7fc3a814f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a80cf9d0 0x7fc3a814f310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 -- 172.21.15.161:0/2443763530 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc398009400 con 0x7fc3a815e400 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a6ffd640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a80cf9d0 0x7fc3a814f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.780+0000 7fc3a77fe640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3a815e400 0x7fc3a80b36f0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc3a000c730 tx=0x7fc3a000cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.781+0000 7fc39f7fe640 1 -- 172.21.15.161:0/2443763530 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a0003da0 con 0x7fc3a815e400 2024-03-23T14:37:31.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.781+0000 7fc39f7fe640 1 -- 172.21.15.161:0/2443763530 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3a000ce40 con 0x7fc3a815e400 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.781+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a80c9ab0 con 0x7fc3a815e400 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.781+0000 7fc39f7fe640 1 -- 172.21.15.161:0/2443763530 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a00052b0 con 0x7fc3a815e400 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.781+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a8159670 con 0x7fc3a815e400 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.782+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3a80ca2b0 con 0x7fc3a815e400 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.782+0000 7fc39f7fe640 1 -- 172.21.15.161:0/2443763530 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc3a0005410 con 0x7fc3a815e400 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.782+0000 7fc39f7fe640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc38806b840 0x7fc38806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.782+0000 7fc39f7fe640 1 -- 172.21.15.161:0/2443763530 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fc3a0089eb0 con 0x7fc3a815e400 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.783+0000 7fc3a6ffd640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc38806b840 0x7fc38806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:31.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.783+0000 7fc3a6ffd640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc38806b840 0x7fc38806dd00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc398036000 tx=0x7fc39803b040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:31.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.787+0000 7fc39f7fe640 1 -- 172.21.15.161:0/2443763530 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc3a80ca2b0 con 0x7fc3a815e400 2024-03-23T14:37:31.962 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.966+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc3a80a1e00 con 0x7fc3a815e400 2024-03-23T14:37:31.963 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.968+0000 7fc39f7fe640 1 -- 172.21.15.161:0/2443763530 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7fc3a0054980 con 0x7fc3a815e400 2024-03-23T14:37:31.964 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:31.964 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:31.965 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.969+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc38806b840 msgr2=0x7fc38806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.969+0000 7fc3ad7f6640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc38806b840 0x7fc38806dd00 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc398036000 tx=0x7fc39803b040 comp rx=0 tx=0).stop 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.969+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3a815e400 msgr2=0x7fc3a80b36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.969+0000 7fc3ad7f6640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3a815e400 0x7fc3a80b36f0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7fc3a000c730 tx=0x7fc3a000cbf0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.970+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 shutdown_connections 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.970+0000 7fc3ad7f6640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc38806b840 0x7fc38806dd00 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.970+0000 7fc3ad7f6640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc3a815e400 0x7fc3a80b36f0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.970+0000 7fc3ad7f6640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc3a80d2b40 0x7fc3a815e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.970+0000 7fc3ad7f6640 1 --2- 172.21.15.161:0/2443763530 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc3a80cf9d0 0x7fc3a814f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.970+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 >> 172.21.15.161:0/2443763530 conn(0x7fc3a809d190 msgr2=0x7fc3a815d5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.970+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 shutdown_connections 2024-03-23T14:37:31.966 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:31.970+0000 7fc3ad7f6640 1 -- 172.21.15.161:0/2443763530 wait complete. 2024-03-23T14:37:31.977 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-23T14:35:00.521058+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-23T14:36:56.747667+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '47', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.75, 'score_stable': 1.75, 'optimal_score': 1, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2024-03-23T14:36:57.478793+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '48', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-23T14:37:31.977 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-23T14:37:32.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.086+0000 7fca5bd43640 1 Processor -- start 2024-03-23T14:37:32.082 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.086+0000 7fca5bd43640 1 -- start start 2024-03-23T14:37:32.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5bd43640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca54161820 0x7fca540d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5bd43640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fca540d2ba0 0x7fca540b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5bd43640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca54161c00 con 0x7fca540d1270 2024-03-23T14:37:32.083 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5bd43640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fca540cea30 con 0x7fca540d2ba0 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5bd43640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fca540cfd70 con 0x7fca54161820 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5a2e0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fca540d2ba0 0x7fca540b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5a2e0640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fca540d2ba0 0x7fca540b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60508/0 (socket says 172.21.15.161:60508) 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca592de640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fca540d1270 0x7fca540d2490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42776/0 (socket says 172.21.15.161:42776) 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5a2e0640 1 -- 172.21.15.161:0/4223706430 learned_addr learned my addr 172.21.15.161:0/4223706430 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca59adf640 1 --2- 172.21.15.161:0/4223706430 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca54161820 0x7fca540d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5a2e0640 1 -- 172.21.15.161:0/4223706430 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca54161820 msgr2=0x7fca540d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5a2e0640 1 --2- 172.21.15.161:0/4223706430 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca54161820 0x7fca540d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5a2e0640 1 -- 172.21.15.161:0/4223706430 >> v1:172.21.15.161:6789/0 conn(0x7fca540d1270 legacy=0x7fca540d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca5a2e0640 1 -- 172.21.15.161:0/4223706430 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca540cecd0 con 0x7fca540d2ba0 2024-03-23T14:37:32.084 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.087+0000 7fca58add640 1 -- 172.21.15.161:0/4223706430 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1410004226 0 0) 0x7fca54161c00 con 0x7fca540d1270 2024-03-23T14:37:32.085 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.088+0000 7fca5a2e0640 1 --2- 172.21.15.161:0/4223706430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca540d2ba0 0x7fca540b0a80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009e50 tx=0x7fca4c02bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67eaddb2b9581fef server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.088+0000 7fca58add640 1 -- 172.21.15.161:0/4223706430 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c02cb00 con 0x7fca540d2ba0 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.088+0000 7fca58add640 1 -- 172.21.15.161:0/4223706430 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca4c02cc60 con 0x7fca540d2ba0 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.088+0000 7fca58add640 1 -- 172.21.15.161:0/4223706430 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c0343f0 con 0x7fca540d2ba0 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.089+0000 7fca5bd43640 1 -- 172.21.15.161:0/4223706430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca540d2ba0 msgr2=0x7fca540b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.089+0000 7fca5bd43640 1 --2- 172.21.15.161:0/4223706430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca540d2ba0 0x7fca540b0a80 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009e50 tx=0x7fca4c02bf60 comp rx=0 tx=0).stop 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.089+0000 7fca5bd43640 1 -- 172.21.15.161:0/4223706430 shutdown_connections 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.089+0000 7fca5bd43640 1 --2- 172.21.15.161:0/4223706430 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca540d2ba0 0x7fca540b0a80 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.089+0000 7fca5bd43640 1 --2- 172.21.15.161:0/4223706430 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca54161820 0x7fca540d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.089+0000 7fca5bd43640 1 -- 172.21.15.161:0/4223706430 >> 172.21.15.161:0/4223706430 conn(0x7fca5409d190 msgr2=0x7fca540d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.089+0000 7fca5bd43640 1 -- 172.21.15.161:0/4223706430 shutdown_connections 2024-03-23T14:37:32.086 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.089+0000 7fca5bd43640 1 -- 172.21.15.161:0/4223706430 wait complete. 2024-03-23T14:37:32.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.091+0000 7fca5bd43640 1 Processor -- start 2024-03-23T14:37:32.087 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.091+0000 7fca5bd43640 1 -- start start 2024-03-23T14:37:32.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5bd43640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca540d1270 0x7fca540ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5bd43640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca540d2ba0 0x7fca541511d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.089 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5bd43640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca54161820 0x7fca540b32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5bd43640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fca540ca860 con 0x7fca54161820 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5bd43640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fca540b98f0 con 0x7fca540d2ba0 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5bd43640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fca540b9610 con 0x7fca540d1270 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5a2e0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca54161820 0x7fca540b32f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5a2e0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca54161820 0x7fca540b32f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60512/0 (socket says 172.21.15.161:60512) 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca5a2e0640 1 -- 172.21.15.161:0/3501617244 learned_addr learned my addr 172.21.15.161:0/3501617244 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca59adf640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca540d1270 0x7fca540ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.092+0000 7fca592de640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca540d2ba0 0x7fca541511d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca5a2e0640 1 -- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca540d1270 msgr2=0x7fca540ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca5a2e0640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca540d1270 0x7fca540ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca5a2e0640 1 -- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca540d2ba0 msgr2=0x7fca541511d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca5a2e0640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca540d2ba0 0x7fca541511d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca5a2e0640 1 -- 172.21.15.161:0/3501617244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca4c009b30 con 0x7fca54161820 2024-03-23T14:37:32.090 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca59adf640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca540d1270 0x7fca540ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:32.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca592de640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca540d2ba0 0x7fca541511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:32.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca5a2e0640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca54161820 0x7fca540b32f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009e50 tx=0x7fca4c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca4a7fc640 1 -- 172.21.15.161:0/3501617244 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c009510 con 0x7fca54161820 2024-03-23T14:37:32.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca4a7fc640 1 -- 172.21.15.161:0/3501617244 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca4c03e450 con 0x7fca54161820 2024-03-23T14:37:32.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca540c7a80 con 0x7fca54161820 2024-03-23T14:37:32.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca4a7fc640 1 -- 172.21.15.161:0/3501617244 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca4c03e600 con 0x7fca54161820 2024-03-23T14:37:32.091 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.093+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca540c8040 con 0x7fca54161820 2024-03-23T14:37:32.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.094+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca5404f4e0 con 0x7fca54161820 2024-03-23T14:37:32.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.094+0000 7fca4a7fc640 1 -- 172.21.15.161:0/3501617244 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fca4c03e760 con 0x7fca54161820 2024-03-23T14:37:32.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.095+0000 7fca4a7fc640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca3806b810 0x7fca3806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.095+0000 7fca4a7fc640 1 -- 172.21.15.161:0/3501617244 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fca4c0acb60 con 0x7fca54161820 2024-03-23T14:37:32.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.095+0000 7fca59adf640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca3806b810 0x7fca3806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.095+0000 7fca59adf640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca3806b810 0x7fca3806dcd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fca50009af0 tx=0x7fca50008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.094 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.099+0000 7fca4a7fc640 1 -- 172.21.15.161:0/3501617244 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fca4c077580 con 0x7fca54161820 2024-03-23T14:37:32.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.348+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fca540b2ba0 con 0x7fca54161820 2024-03-23T14:37:32.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.348+0000 7fca4a7fc640 1 -- 172.21.15.161:0/3501617244 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v48) ==== 93+0+10 (secure 0 0 0) 0x7fca4c07ac40 con 0x7fca54161820 2024-03-23T14:37:32.344 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 1 2024-03-23T14:37:32.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca3806b810 msgr2=0x7fca3806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.346 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca3806b810 0x7fca3806dcd0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fca50009af0 tx=0x7fca50008040 comp rx=0 tx=0).stop 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca54161820 msgr2=0x7fca540b32f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca54161820 0x7fca540b32f0 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7fca4c009e50 tx=0x7fca4c004a40 comp rx=0 tx=0).stop 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 shutdown_connections 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fca3806b810 0x7fca3806dcd0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fca54161820 0x7fca540b32f0 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fca540d2ba0 0x7fca541511d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 --2- 172.21.15.161:0/3501617244 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fca540d1270 0x7fca540ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 >> 172.21.15.161:0/3501617244 conn(0x7fca5409d190 msgr2=0x7fca540b23b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 shutdown_connections 2024-03-23T14:37:32.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.350+0000 7fca5bd43640 1 -- 172.21.15.161:0/3501617244 wait complete. 2024-03-23T14:37:32.359 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-23T14:37:32.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.468+0000 7f5a89db5640 1 Processor -- start 2024-03-23T14:37:32.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.468+0000 7f5a89db5640 1 -- start start 2024-03-23T14:37:32.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a89db5640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a840ce5e0 0x7f5a840d1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a89db5640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a840b3710 con 0x7f5a8415de70 2024-03-23T14:37:32.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a89db5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a8415ecd0 con 0x7f5a840d1f60 2024-03-23T14:37:32.465 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a89db5640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5a840d23d0 con 0x7f5a840ce5e0 2024-03-23T14:37:32.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a7affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a840ce5e0 0x7f5a840d1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a7affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5a840ce5e0 0x7f5a840d1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60522/0 (socket says 172.21.15.161:60522) 2024-03-23T14:37:32.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a7affd640 1 -- 172.21.15.161:0/634362667 learned_addr learned my addr 172.21.15.161:0/634362667 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:32.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a837fe640 1 --1- 172.21.15.161:0/634362667 >> v1:172.21.15.161:6789/0 conn(0x7f5a8415de70 0x7f5a8415e250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42788/0 (socket says 172.21.15.161:42788) 2024-03-23T14:37:32.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a7affd640 1 -- 172.21.15.161:0/634362667 >> v1:172.21.15.183:6789/0 conn(0x7f5a840d1f60 legacy=0x7f5a8406c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:32.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a7affd640 1 -- 172.21.15.161:0/634362667 >> v1:172.21.15.161:6789/0 conn(0x7f5a8415de70 legacy=0x7f5a8415e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.469+0000 7f5a7affd640 1 -- 172.21.15.161:0/634362667 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a8415efb0 con 0x7f5a840ce5e0 2024-03-23T14:37:32.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.470+0000 7f5a7affd640 1 --2- 172.21.15.161:0/634362667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a840ce5e0 0x7f5a840d1a20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009870 tx=0x7f5a6c02b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98e98246b12c65eb server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.470+0000 7f5a82ffd640 1 -- 172.21.15.161:0/634362667 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c030850 con 0x7f5a840ce5e0 2024-03-23T14:37:32.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.470+0000 7f5a82ffd640 1 -- 172.21.15.161:0/634362667 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a6c0309b0 con 0x7f5a840ce5e0 2024-03-23T14:37:32.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.470+0000 7f5a82ffd640 1 -- 172.21.15.161:0/634362667 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a6c030d40 con 0x7f5a840ce5e0 2024-03-23T14:37:32.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.471+0000 7f5a89db5640 1 -- 172.21.15.161:0/634362667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a840ce5e0 msgr2=0x7f5a840d1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.471+0000 7f5a89db5640 1 --2- 172.21.15.161:0/634362667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a840ce5e0 0x7f5a840d1a20 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f5a6c009870 tx=0x7f5a6c02b990 comp rx=0 tx=0).stop 2024-03-23T14:37:32.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.471+0000 7f5a89db5640 1 -- 172.21.15.161:0/634362667 shutdown_connections 2024-03-23T14:37:32.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.471+0000 7f5a89db5640 1 --2- 172.21.15.161:0/634362667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a840ce5e0 0x7f5a840d1a20 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.471+0000 7f5a89db5640 1 -- 172.21.15.161:0/634362667 >> 172.21.15.161:0/634362667 conn(0x7f5a8409aa40 msgr2=0x7f5a8409ae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:32.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.471+0000 7f5a89db5640 1 -- 172.21.15.161:0/634362667 shutdown_connections 2024-03-23T14:37:32.468 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.471+0000 7f5a89db5640 1 -- 172.21.15.161:0/634362667 wait complete. 2024-03-23T14:37:32.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.472+0000 7f5a89db5640 1 Processor -- start 2024-03-23T14:37:32.470 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.472+0000 7f5a89db5640 1 -- start start 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a89db5640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a840ce5e0 0x7f5a841e7a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a89db5640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a840d1f60 0x7f5a840b6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a89db5640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a8415de70 0x7f5a84159300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a89db5640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5a841e8d30 con 0x7f5a8415de70 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a89db5640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5a840b6ff0 con 0x7f5a840ce5e0 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a89db5640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5a84159840 con 0x7f5a840d1f60 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a83fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a8415de70 0x7f5a84159300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a83fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a8415de70 0x7f5a84159300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60530/0 (socket says 172.21.15.161:60530) 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a83fff640 1 -- 172.21.15.161:0/3932004667 learned_addr learned my addr 172.21.15.161:0/3932004667 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a837fe640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a840ce5e0 0x7f5a841e7a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a7affd640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a840d1f60 0x7f5a840b6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a83fff640 1 -- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a840d1f60 msgr2=0x7f5a840b6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a83fff640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a840d1f60 0x7f5a840b6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.471 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a83fff640 1 -- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a840ce5e0 msgr2=0x7f5a841e7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a83fff640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a840ce5e0 0x7f5a841e7a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.473+0000 7f5a83fff640 1 -- 172.21.15.161:0/3932004667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a6c0094d0 con 0x7f5a8415de70 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.474+0000 7f5a83fff640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a8415de70 0x7f5a84159300 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00c780 tx=0x7f5a7c00cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.474+0000 7f5a80ff9640 1 -- 172.21.15.161:0/3932004667 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c004b60 con 0x7f5a8415de70 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.474+0000 7f5a80ff9640 1 -- 172.21.15.161:0/3932004667 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a7c00ce90 con 0x7f5a8415de70 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.474+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a841e80b0 con 0x7f5a8415de70 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.474+0000 7f5a80ff9640 1 -- 172.21.15.161:0/3932004667 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a7c00a850 con 0x7f5a8415de70 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.474+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a841e8670 con 0x7f5a8415de70 2024-03-23T14:37:32.472 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.475+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a840b7c70 con 0x7f5a8415de70 2024-03-23T14:37:32.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.476+0000 7f5a80ff9640 1 -- 172.21.15.161:0/3932004667 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5a7c00a9b0 con 0x7f5a8415de70 2024-03-23T14:37:32.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.476+0000 7f5a80ff9640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a6406b810 0x7f5a6406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.476+0000 7f5a80ff9640 1 -- 172.21.15.161:0/3932004667 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5a7c08a370 con 0x7f5a8415de70 2024-03-23T14:37:32.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.476+0000 7f5a837fe640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a6406b810 0x7f5a6406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.473 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.477+0000 7f5a837fe640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a6406b810 0x7f5a6406dcd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5a74007fd0 tx=0x7f5a74007e80 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.480+0000 7f5a80ff9640 1 -- 172.21.15.161:0/3932004667 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5a7c054d10 con 0x7f5a8415de70 2024-03-23T14:37:32.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.659+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5a840b2c90 con 0x7f5a8415de70 2024-03-23T14:37:32.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.659+0000 7f5a80ff9640 1 -- 172.21.15.161:0/3932004667 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v48) ==== 104+0+11 (secure 0 0 0) 0x7f5a7c0583d0 con 0x7f5a8415de70 2024-03-23T14:37:32.655 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 32 2024-03-23T14:37:32.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a6406b810 msgr2=0x7f5a6406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a6406b810 0x7f5a6406dcd0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5a74007fd0 tx=0x7f5a74007e80 comp rx=0 tx=0).stop 2024-03-23T14:37:32.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a8415de70 msgr2=0x7f5a84159300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a8415de70 0x7f5a84159300 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f5a7c00c780 tx=0x7f5a7c00cc40 comp rx=0 tx=0).stop 2024-03-23T14:37:32.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 shutdown_connections 2024-03-23T14:37:32.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5a6406b810 0x7f5a6406dcd0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5a8415de70 0x7f5a84159300 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5a840d1f60 0x7f5a840b6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 --2- 172.21.15.161:0/3932004667 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5a840ce5e0 0x7f5a841e7a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 >> 172.21.15.161:0/3932004667 conn(0x7f5a8409aa40 msgr2=0x7f5a8406eff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:32.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 shutdown_connections 2024-03-23T14:37:32.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.661+0000 7f5a89db5640 1 -- 172.21.15.161:0/3932004667 wait complete. 2024-03-23T14:37:32.668 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-23T14:37:32.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.777+0000 7efff25c7640 1 Processor -- start 2024-03-23T14:37:32.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.777+0000 7efff25c7640 1 -- start start 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7efff25c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7effec161820 0x7effec0d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7efff25c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d1270 0x7effec0d2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7efff25c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effec161c00 con 0x7effec0d29d0 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7efff25c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7effec0d2e20 con 0x7effec161820 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7efff25c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7effec0cea30 con 0x7effec0d1270 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7efff0b64640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7effec0d29d0 0x7effec161e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42794/0 (socket says 172.21.15.161:42794) 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7efff0b64640 1 -- 172.21.15.161:0/639934386 learned_addr learned my addr 172.21.15.161:0/639934386 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7effebfff640 1 --2- 172.21.15.161:0/639934386 >> v2:172.21.15.183:3300/0 conn(0x7effec161820 0x7effec0d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.778+0000 7effeb7fe640 1 --2- 172.21.15.161:0/639934386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d1270 0x7effec0d2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.776 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.779+0000 7effeaffd640 1 -- 172.21.15.161:0/639934386 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2179737008 0 0) 0x7effec161c00 con 0x7effec0d29d0 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.779+0000 7effeaffd640 1 -- 172.21.15.161:0/639934386 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effd4003520 con 0x7effec0d29d0 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.779+0000 7effebfff640 1 -- 172.21.15.161:0/639934386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d1270 msgr2=0x7effec0d2490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.779+0000 7effebfff640 1 --2- 172.21.15.161:0/639934386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d1270 0x7effec0d2490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.779+0000 7effebfff640 1 -- 172.21.15.161:0/639934386 >> v1:172.21.15.161:6789/0 conn(0x7effec0d29d0 legacy=0x7effec161e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.779+0000 7effebfff640 1 -- 172.21.15.161:0/639934386 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effec0cfd70 con 0x7effec161820 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.780+0000 7effebfff640 1 --2- 172.21.15.161:0/639934386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec161820 0x7effec0d47e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7effe00097a0 tx=0x7effe002b8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b81c797e4a69a41c server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.780+0000 7effeaffd640 1 -- 172.21.15.161:0/639934386 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0031830 con 0x7effec161820 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.780+0000 7effeaffd640 1 -- 172.21.15.161:0/639934386 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe0031990 con 0x7effec161820 2024-03-23T14:37:32.777 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.780+0000 7effeaffd640 1 -- 172.21.15.161:0/639934386 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0031d20 con 0x7effec161820 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.781+0000 7efff25c7640 1 -- 172.21.15.161:0/639934386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec161820 msgr2=0x7effec0d47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.781+0000 7efff25c7640 1 --2- 172.21.15.161:0/639934386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec161820 0x7effec0d47e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7effe00097a0 tx=0x7effe002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.781+0000 7efff25c7640 1 -- 172.21.15.161:0/639934386 shutdown_connections 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.781+0000 7efff25c7640 1 --2- 172.21.15.161:0/639934386 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d1270 0x7effec0d2490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.781+0000 7efff25c7640 1 --2- 172.21.15.161:0/639934386 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec161820 0x7effec0d47e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.781+0000 7efff25c7640 1 -- 172.21.15.161:0/639934386 >> 172.21.15.161:0/639934386 conn(0x7effec09d190 msgr2=0x7effec0d3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.781+0000 7efff25c7640 1 -- 172.21.15.161:0/639934386 shutdown_connections 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.781+0000 7efff25c7640 1 -- 172.21.15.161:0/639934386 wait complete. 2024-03-23T14:37:32.779 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.782+0000 7efff25c7640 1 Processor -- start 2024-03-23T14:37:32.780 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.783+0000 7efff25c7640 1 -- start start 2024-03-23T14:37:32.782 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff25c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec0d1270 0x7effec0b6aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff25c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d29d0 0x7effec0b4590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff25c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effec161820 0x7effec0cbea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff25c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7effec0caf80 con 0x7effec161820 2024-03-23T14:37:32.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff25c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7effec0cb1d0 con 0x7effec0d1270 2024-03-23T14:37:32.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff25c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7effec0cb4b0 con 0x7effec0d29d0 2024-03-23T14:37:32.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff0b64640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effec161820 0x7effec0cbea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff0b64640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effec161820 0x7effec0cbea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60546/0 (socket says 172.21.15.161:60546) 2024-03-23T14:37:32.783 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff0b64640 1 -- 172.21.15.161:0/298895691 learned_addr learned my addr 172.21.15.161:0/298895691 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7effebfff640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec0d1270 0x7effec0b6aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7effeb7fe640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d29d0 0x7effec0b4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff0b64640 1 -- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d29d0 msgr2=0x7effec0b4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff0b64640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d29d0 0x7effec0b4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff0b64640 1 -- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec0d1270 msgr2=0x7effec0b6aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff0b64640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec0d1270 0x7effec0b6aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.784+0000 7efff0b64640 1 -- 172.21.15.161:0/298895691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe0009400 con 0x7effec161820 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.785+0000 7efff0b64640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effec161820 0x7effec0cbea0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7effe400cb40 tx=0x7effe400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.785+0000 7effe8ff9640 1 -- 172.21.15.161:0/298895691 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe400aca0 con 0x7effec161820 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.785+0000 7effe8ff9640 1 -- 172.21.15.161:0/298895691 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effe400ae00 con 0x7effec161820 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.785+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effec0cb790 con 0x7effec161820 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.785+0000 7effe8ff9640 1 -- 172.21.15.161:0/298895691 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe4012620 con 0x7effec161820 2024-03-23T14:37:32.784 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.785+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effec0b4ad0 con 0x7effec161820 2024-03-23T14:37:32.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.786+0000 7effe8ff9640 1 -- 172.21.15.161:0/298895691 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7effe4007070 con 0x7effec161820 2024-03-23T14:37:32.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.786+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effec0c79e0 con 0x7effec161820 2024-03-23T14:37:32.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.787+0000 7effe8ff9640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7effd806b840 0x7effd806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:32.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.787+0000 7effe8ff9640 1 -- 172.21.15.161:0/298895691 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7effe408a1c0 con 0x7effec161820 2024-03-23T14:37:32.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.787+0000 7effebfff640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7effd806b840 0x7effd806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:32.785 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.787+0000 7effebfff640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7effd806b840 0x7effd806dd00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7effe0036000 tx=0x7effe003c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:32.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:32.791+0000 7effe8ff9640 1 -- 172.21.15.161:0/298895691 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7effe4054c70 con 0x7effec161820 2024-03-23T14:37:33.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.124+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7effec04f4e0 con 0x7effec161820 2024-03-23T14:37:33.121 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.125+0000 7effe8ff9640 1 -- 172.21.15.161:0/298895691 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v48) ==== 100+0+11 (secure 0 0 0) 0x7effe4058330 con 0x7effec161820 2024-03-23T14:37:33.121 INFO:teuthology.orchestra.run.smithi161.stdout:pg_num: 64 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7effd806b840 msgr2=0x7effd806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7effd806b840 0x7effd806dd00 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7effe0036000 tx=0x7effe003c040 comp rx=0 tx=0).stop 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effec161820 msgr2=0x7effec0cbea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effec161820 0x7effec0cbea0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7effe400cb40 tx=0x7effe400a060 comp rx=0 tx=0).stop 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 shutdown_connections 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7effd806b840 0x7effd806dd00 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7effec161820 0x7effec0cbea0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7effec0d29d0 0x7effec0b4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 --2- 172.21.15.161:0/298895691 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7effec0d1270 0x7effec0b6aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 >> 172.21.15.161:0/298895691 conn(0x7effec09d190 msgr2=0x7effec0b10a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:33.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 shutdown_connections 2024-03-23T14:37:33.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.127+0000 7efff25c7640 1 -- 172.21.15.161:0/298895691 wait complete. 2024-03-23T14:37:33.134 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:37:33.240 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.244+0000 7f5b76b02640 1 Processor -- start 2024-03-23T14:37:33.241 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.245+0000 7f5b76b02640 1 -- start start 2024-03-23T14:37:33.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.245+0000 7f5b76b02640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f5b7015e060 0x7f5b700727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.245+0000 7f5b76b02640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b700d14c0 0x7f5b7006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.245+0000 7f5b76b02640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5b7015e440 con 0x7f5b700b5b80 2024-03-23T14:37:33.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.245+0000 7f5b76b02640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5b700b0480 con 0x7f5b700d14c0 2024-03-23T14:37:33.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.245+0000 7f5b76b02640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f5b700b5fe0 con 0x7f5b7015e060 2024-03-23T14:37:33.242 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.245+0000 7f5b7509f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b700d14c0 0x7f5b7006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.246+0000 7f5b6ffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5b700b5b80 0x7f5b700d0db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42804/0 (socket says 172.21.15.161:42804) 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.246+0000 7f5b7509f640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5b700d14c0 0x7f5b7006dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60550/0 (socket says 172.21.15.161:60550) 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.246+0000 7f5b6ffff640 1 -- 172.21.15.161:0/3486268836 learned_addr learned my addr 172.21.15.161:0/3486268836 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.246+0000 7f5b7489e640 1 --2- 172.21.15.161:0/3486268836 >> v2:172.21.15.183:3300/0 conn(0x7f5b7015e060 0x7f5b700727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.246+0000 7f5b7509f640 1 -- 172.21.15.161:0/3486268836 >> v2:172.21.15.183:3300/0 conn(0x7f5b7015e060 msgr2=0x7f5b700727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.246+0000 7f5b7509f640 1 --2- 172.21.15.161:0/3486268836 >> v2:172.21.15.183:3300/0 conn(0x7f5b7015e060 0x7f5b700727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.246+0000 7f5b7509f640 1 -- 172.21.15.161:0/3486268836 >> v1:172.21.15.161:6789/0 conn(0x7f5b700b5b80 legacy=0x7f5b700d0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.246+0000 7f5b7509f640 1 -- 172.21.15.161:0/3486268836 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b70161cf0 con 0x7f5b700d14c0 2024-03-23T14:37:33.243 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.247+0000 7f5b7509f640 1 --2- 172.21.15.161:0/3486268836 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b7006dcd0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f5b700d1980 tx=0x7f5b6802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=502385bd57b5fa14 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.248+0000 7f5b6f7fe640 1 -- 172.21.15.161:0/3486268836 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b6802cc80 con 0x7f5b700d14c0 2024-03-23T14:37:33.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.248+0000 7f5b6f7fe640 1 -- 172.21.15.161:0/3486268836 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b6802cde0 con 0x7f5b700d14c0 2024-03-23T14:37:33.244 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.248+0000 7f5b6f7fe640 1 -- 172.21.15.161:0/3486268836 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b68034660 con 0x7f5b700d14c0 2024-03-23T14:37:33.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.249+0000 7f5b76b02640 1 -- 172.21.15.161:0/3486268836 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 msgr2=0x7f5b7006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.249+0000 7f5b76b02640 1 --2- 172.21.15.161:0/3486268836 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b7006dcd0 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f5b700d1980 tx=0x7f5b6802bed0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.249+0000 7f5b76b02640 1 -- 172.21.15.161:0/3486268836 shutdown_connections 2024-03-23T14:37:33.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.249+0000 7f5b76b02640 1 --2- 172.21.15.161:0/3486268836 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b7006dcd0 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.249+0000 7f5b76b02640 1 --2- 172.21.15.161:0/3486268836 >> v2:172.21.15.183:3300/0 conn(0x7f5b7015e060 0x7f5b700727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.249+0000 7f5b76b02640 1 -- 172.21.15.161:0/3486268836 >> 172.21.15.161:0/3486268836 conn(0x7f5b7006c370 msgr2=0x7f5b7006c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:33.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.249+0000 7f5b76b02640 1 -- 172.21.15.161:0/3486268836 shutdown_connections 2024-03-23T14:37:33.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.249+0000 7f5b76b02640 1 -- 172.21.15.161:0/3486268836 wait complete. 2024-03-23T14:37:33.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.250+0000 7f5b76b02640 1 Processor -- start 2024-03-23T14:37:33.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.250+0000 7f5b76b02640 1 -- start start 2024-03-23T14:37:33.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b76b02640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b700b5b80 0x7f5b700b0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b76b02640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b70152300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b76b02640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b7015e060 0x7f5b70160c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b76b02640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7015f990 con 0x7f5b700d14c0 2024-03-23T14:37:33.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b76b02640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5b7015fb70 con 0x7f5b7015e060 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b76b02640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5b7015fe50 con 0x7f5b700b5b80 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b6ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b70152300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b6ffff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b70152300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60560/0 (socket says 172.21.15.161:60560) 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b6ffff640 1 -- 172.21.15.161:0/1755498063 learned_addr learned my addr 172.21.15.161:0/1755498063 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b7489e640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b700b5b80 0x7f5b700b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.251+0000 7f5b7509f640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b7015e060 0x7f5b70160c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6ffff640 1 -- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b700b5b80 msgr2=0x7f5b700b0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6ffff640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b700b5b80 0x7f5b700b0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6ffff640 1 -- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b7015e060 msgr2=0x7f5b70160c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6ffff640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b7015e060 0x7f5b70160c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6ffff640 1 -- 172.21.15.161:0/1755498063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b68009a70 con 0x7f5b700d14c0 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6ffff640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b70152300 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00c400 tx=0x7f5b5c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6d7fa640 1 -- 172.21.15.161:0/1755498063 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c005ce0 con 0x7f5b700d14c0 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6d7fa640 1 -- 172.21.15.161:0/1755498063 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5b5c005e40 con 0x7f5b700d14c0 2024-03-23T14:37:33.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b70160130 con 0x7f5b700d14c0 2024-03-23T14:37:33.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b6d7fa640 1 -- 172.21.15.161:0/1755498063 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b5c007310 con 0x7f5b700d14c0 2024-03-23T14:37:33.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.252+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b701606f0 con 0x7f5b700d14c0 2024-03-23T14:37:33.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.253+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b70073680 con 0x7f5b700d14c0 2024-03-23T14:37:33.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.253+0000 7f5b6d7fa640 1 -- 172.21.15.161:0/1755498063 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5b5c0074f0 con 0x7f5b700d14c0 2024-03-23T14:37:33.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.254+0000 7f5b6d7fa640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5406b810 0x7f5b5406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.254+0000 7f5b6d7fa640 1 -- 172.21.15.161:0/1755498063 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5b5c089680 con 0x7f5b700d14c0 2024-03-23T14:37:33.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.254+0000 7f5b7489e640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5406b810 0x7f5b5406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.255+0000 7f5b7489e640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5406b810 0x7f5b5406dcd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009aa0 tx=0x7f5b64009190 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.258+0000 7f5b6d7fa640 1 -- 172.21.15.161:0/1755498063 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5b5c0541e0 con 0x7f5b700d14c0 2024-03-23T14:37:33.463 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.467+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5b7004f550 con 0x7f5b700d14c0 2024-03-23T14:37:33.464 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.468+0000 7f5b6d7fa640 1 -- 172.21.15.161:0/1755498063 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5193 (secure 0 0 0) 0x7f5b5c0578a0 con 0x7f5b700d14c0 2024-03-23T14:37:33.464 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:33.464 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2496234294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":2496234294},{"type":"v1","addr":"172.21.15.183:6836","nonce":2496234294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5396,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2645821131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2645821131},{"type":"v1","addr":"172.21.15.183:6837","nonce":2645821131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/2183619206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2183619206},{"type":"v1","addr":"172.21.15.161:6834","nonce":2183619206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:58.503230+0000","modified":"2024-03-23T14:37:00.601296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/1671955402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":1671955402},{"type":"v1","addr":"172.21.15.161:6835","nonce":1671955402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5925,"qdb_cluster":[5925]},"id":3}]} 2024-03-23T14:37:33.464 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-23T14:37:33.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5406b810 msgr2=0x7f5b5406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5406b810 0x7f5b5406dcd0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f5b64009aa0 tx=0x7f5b64009190 comp rx=0 tx=0).stop 2024-03-23T14:37:33.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 msgr2=0x7f5b70152300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b70152300 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f5b5c00c400 tx=0x7f5b5c00c8c0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 shutdown_connections 2024-03-23T14:37:33.466 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5b5406b810 0x7f5b5406dcd0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5b7015e060 0x7f5b70160c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5b700d14c0 0x7f5b70152300 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 --2- 172.21.15.161:0/1755498063 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5b700b5b80 0x7f5b700b0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 >> 172.21.15.161:0/1755498063 conn(0x7f5b7006c370 msgr2=0x7f5b70154100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:33.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 shutdown_connections 2024-03-23T14:37:33.467 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.470+0000 7f5b76b02640 1 -- 172.21.15.161:0/1755498063 wait complete. 2024-03-23T14:37:33.477 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:33.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.583+0000 7feba9525640 1 Processor -- start 2024-03-23T14:37:33.579 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.583+0000 7feba9525640 1 -- start start 2024-03-23T14:37:33.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feba9525640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba40cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feba9525640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feba40cf9d0 0x7feba40d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feba9525640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7feba40d1470 0x7feba40d2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.582 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feba9525640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7feba40d1050 con 0x7feba40cf9d0 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feba9525640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7feba40d0460 con 0x7feba40d1470 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feba9525640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feba40d06b0 con 0x7feba415e400 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feb9bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feba40cf9d0 0x7feba40d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feb9bfff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7feba40cf9d0 0x7feba40d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60564/0 (socket says 172.21.15.161:60564) 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.584+0000 7feb9bfff640 1 -- 172.21.15.161:0/2789726347 learned_addr learned my addr 172.21.15.161:0/2789726347 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feba37fe640 1 --2- 172.21.15.161:0/2789726347 >> v2:172.21.15.183:3300/0 conn(0x7feba40d1470 0x7feba40d2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feba2ffd640 1 --2- 172.21.15.161:0/2789726347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba40cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feb9bfff640 1 -- 172.21.15.161:0/2789726347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 msgr2=0x7feba40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feb9bfff640 1 --2- 172.21.15.161:0/2789726347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba40cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feb9bfff640 1 -- 172.21.15.161:0/2789726347 >> v2:172.21.15.183:3300/0 conn(0x7feba40d1470 msgr2=0x7feba40d2970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feb9bfff640 1 --2- 172.21.15.161:0/2789726347 >> v2:172.21.15.183:3300/0 conn(0x7feba40d1470 0x7feba40d2970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feb9bfff640 1 -- 172.21.15.161:0/2789726347 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba40d0990 con 0x7feba40cf9d0 2024-03-23T14:37:33.583 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feba2ffd640 1 --2- 172.21.15.161:0/2789726347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.585+0000 7feb9bfff640 1 --2- 172.21.15.161:0/2789726347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40cf9d0 0x7feba40d2430 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7feba408e3f0 tx=0x7feb8c02b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24ee7a1ab56b8457 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba27fc640 1 -- 172.21.15.161:0/2789726347 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c031870 con 0x7feba40cf9d0 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba27fc640 1 -- 172.21.15.161:0/2789726347 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb8c0319d0 con 0x7feba40cf9d0 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba27fc640 1 -- 172.21.15.161:0/2789726347 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c031d60 con 0x7feba40cf9d0 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 -- 172.21.15.161:0/2789726347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40cf9d0 msgr2=0x7feba40d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 --2- 172.21.15.161:0/2789726347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40cf9d0 0x7feba40d2430 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7feba408e3f0 tx=0x7feb8c02b910 comp rx=0 tx=0).stop 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 -- 172.21.15.161:0/2789726347 shutdown_connections 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 --2- 172.21.15.161:0/2789726347 >> v2:172.21.15.183:3300/0 conn(0x7feba40d1470 0x7feba40d2970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 --2- 172.21.15.161:0/2789726347 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40cf9d0 0x7feba40d2430 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 --2- 172.21.15.161:0/2789726347 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba40cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 -- 172.21.15.161:0/2789726347 >> 172.21.15.161:0/2789726347 conn(0x7feba409d190 msgr2=0x7feba40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 -- 172.21.15.161:0/2789726347 shutdown_connections 2024-03-23T14:37:33.584 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.586+0000 7feba9525640 1 -- 172.21.15.161:0/2789726347 wait complete. 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.587+0000 7feba9525640 1 Processor -- start 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.587+0000 7feba9525640 1 -- start start 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feba9525640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feba40cf9d0 0x7feba41e85b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feba9525640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40d1470 0x7feba40b5c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feba9525640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba415a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feba9525640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7feba40b7d20 con 0x7feba40d1470 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feba9525640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7feba41e8ec0 con 0x7feba40cf9d0 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feba9525640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7feba40b85a0 con 0x7feba415e400 2024-03-23T14:37:33.588 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feb9bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40d1470 0x7feba40b5c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feb9bfff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40d1470 0x7feba40b5c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60568/0 (socket says 172.21.15.161:60568) 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feb9bfff640 1 -- 172.21.15.161:0/1609962960 learned_addr learned my addr 172.21.15.161:0/1609962960 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feba2ffd640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feba40cf9d0 0x7feba41e85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.588+0000 7feba37fe640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba415a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9bfff640 1 -- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 msgr2=0x7feba415a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9bfff640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba415a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9bfff640 1 -- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feba40cf9d0 msgr2=0x7feba41e85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9bfff640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feba40cf9d0 0x7feba41e85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9bfff640 1 -- 172.21.15.161:0/1609962960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb8c0094d0 con 0x7feba40d1470 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9bfff640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40d1470 0x7feba40b5c40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0097c0 tx=0x7feb8c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9b7fe640 1 -- 172.21.15.161:0/1609962960 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c03ac60 con 0x7feba40d1470 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9b7fe640 1 -- 172.21.15.161:0/1609962960 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb8c03adc0 con 0x7feba40d1470 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba415a760 con 0x7feba40d1470 2024-03-23T14:37:33.589 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feb9b7fe640 1 -- 172.21.15.161:0/1609962960 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb8c044610 con 0x7feba40d1470 2024-03-23T14:37:33.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.589+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba40b7f50 con 0x7feba40d1470 2024-03-23T14:37:33.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.590+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feba40d3e10 con 0x7feba40d1470 2024-03-23T14:37:33.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.590+0000 7feb9b7fe640 1 -- 172.21.15.161:0/1609962960 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7feb8c04d450 con 0x7feba40d1470 2024-03-23T14:37:33.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.591+0000 7feb9b7fe640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb8406b810 0x7feb8406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.591+0000 7feb9b7fe640 1 -- 172.21.15.161:0/1609962960 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7feb8c02d040 con 0x7feba40d1470 2024-03-23T14:37:33.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.591+0000 7feba2ffd640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb8406b810 0x7feb8406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.590 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.592+0000 7feba2ffd640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb8406b810 0x7feb8406dcd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feb9400a950 tx=0x7feb94008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.595+0000 7feb9b7fe640 1 -- 172.21.15.161:0/1609962960 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7feb8c0772e0 con 0x7feba40d1470 2024-03-23T14:37:33.769 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.773+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feba40a1e00 con 0x7feba40d1470 2024-03-23T14:37:33.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.775+0000 7feb9b7fe640 1 -- 172.21.15.161:0/1609962960 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7feb8c035020 con 0x7feba40d1470 2024-03-23T14:37:33.771 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:33.771 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.776+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb8406b810 msgr2=0x7feb8406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.776+0000 7feba9525640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb8406b810 0x7feb8406dcd0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7feb9400a950 tx=0x7feb94008040 comp rx=0 tx=0).stop 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40d1470 msgr2=0x7feba40b5c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40d1470 0x7feba40b5c40 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7feb8c0097c0 tx=0x7feb8c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 shutdown_connections 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7feb8406b810 0x7feb8406dcd0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7feba415e400 0x7feba415a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7feba40d1470 0x7feba40b5c40 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 --2- 172.21.15.161:0/1609962960 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7feba40cf9d0 0x7feba41e85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 >> 172.21.15.161:0/1609962960 conn(0x7feba409d190 msgr2=0x7feba415ee30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:33.773 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 shutdown_connections 2024-03-23T14:37:33.774 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.777+0000 7feba9525640 1 -- 172.21.15.161:0/1609962960 wait complete. 2024-03-23T14:37:33.784 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T14:37:33.784 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:37:33.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.900+0000 7f5d756ed640 1 Processor -- start 2024-03-23T14:37:33.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.900+0000 7f5d756ed640 1 -- start start 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.900+0000 7f5d756ed640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5d700d1470 0x7f5d700d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.900+0000 7f5d756ed640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d700d0460 con 0x7f5d7015e400 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.900+0000 7f5d756ed640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d700d06b0 con 0x7f5d700d2970 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.900+0000 7f5d756ed640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f5d700d1050 con 0x7f5d700d1470 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.901+0000 7f5d67fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f5d700d1470 0x7f5d700d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.901+0000 7f5d6effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5d7015e400 0x7f5d700cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42808/0 (socket says 172.21.15.161:42808) 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.901+0000 7f5d6effd640 1 -- 172.21.15.161:0/3967393697 learned_addr learned my addr 172.21.15.161:0/3967393697 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.901+0000 7f5d67fff640 1 -- 172.21.15.161:0/3967393697 >> v1:172.21.15.183:6789/0 conn(0x7f5d700d2970 legacy=0x7f5d700b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.901+0000 7f5d67fff640 1 -- 172.21.15.161:0/3967393697 >> v1:172.21.15.161:6789/0 conn(0x7f5d7015e400 legacy=0x7f5d700cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.897 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.901+0000 7f5d6e7fc640 1 -- 172.21.15.161:0/3967393697 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 519841389 0 0) 0x7f5d700d0460 con 0x7f5d7015e400 2024-03-23T14:37:33.898 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.901+0000 7f5d67fff640 1 -- 172.21.15.161:0/3967393697 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d700d0990 con 0x7f5d700d1470 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.902+0000 7f5d67fff640 1 --2- 172.21.15.161:0/3967393697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d700d2430 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5d580097a0 tx=0x7f5d5802b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e681da379437391f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.902+0000 7f5d6e7fc640 1 -- 172.21.15.161:0/3967393697 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58031830 con 0x7f5d700d1470 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.902+0000 7f5d6e7fc640 1 -- 172.21.15.161:0/3967393697 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d58031990 con 0x7f5d700d1470 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.902+0000 7f5d6e7fc640 1 -- 172.21.15.161:0/3967393697 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d58031d20 con 0x7f5d700d1470 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.903+0000 7f5d756ed640 1 -- 172.21.15.161:0/3967393697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 msgr2=0x7f5d700d2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.903+0000 7f5d756ed640 1 --2- 172.21.15.161:0/3967393697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d700d2430 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f5d580097a0 tx=0x7f5d5802b8d0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.903+0000 7f5d756ed640 1 -- 172.21.15.161:0/3967393697 shutdown_connections 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.903+0000 7f5d756ed640 1 --2- 172.21.15.161:0/3967393697 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d700d2430 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.899 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.903+0000 7f5d756ed640 1 -- 172.21.15.161:0/3967393697 >> 172.21.15.161:0/3967393697 conn(0x7f5d7009d190 msgr2=0x7f5d700d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:33.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.903+0000 7f5d756ed640 1 -- 172.21.15.161:0/3967393697 shutdown_connections 2024-03-23T14:37:33.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.903+0000 7f5d756ed640 1 -- 172.21.15.161:0/3967393697 wait complete. 2024-03-23T14:37:33.900 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.905+0000 7f5d756ed640 1 Processor -- start 2024-03-23T14:37:33.901 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.905+0000 7f5d756ed640 1 -- start start 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d756ed640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d70158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d756ed640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d700d2970 0x7f5d700b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d756ed640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d7015e400 0x7f5d700b4550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d756ed640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5d700b8ef0 con 0x7f5d700d1470 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d756ed640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5d701f4a50 con 0x7f5d700d2970 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d756ed640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5d701f4180 con 0x7f5d7015e400 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d70158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d70158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60582/0 (socket says 172.21.15.161:60582) 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 -- 172.21.15.161:0/3021491842 learned_addr learned my addr 172.21.15.161:0/3021491842 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 -- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d7015e400 msgr2=0x7f5d700b4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.903 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d7015e400 0x7f5d700b4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 -- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d700d2970 msgr2=0x7f5d700b81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d700d2970 0x7f5d700b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 -- 172.21.15.161:0/3021491842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d58009400 con 0x7f5d700d1470 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.906+0000 7f5d6effd640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d70158f80 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c500 tx=0x7f5d6000c9c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.907+0000 7f5d677fe640 1 -- 172.21.15.161:0/3021491842 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60007560 con 0x7f5d700d1470 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.907+0000 7f5d677fe640 1 -- 172.21.15.161:0/3021491842 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5d600047e0 con 0x7f5d700d1470 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.907+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d700b8c10 con 0x7f5d700d1470 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.907+0000 7f5d677fe640 1 -- 172.21.15.161:0/3021491842 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d60004b90 con 0x7f5d700d1470 2024-03-23T14:37:33.904 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.907+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d701f46e0 con 0x7f5d700d1470 2024-03-23T14:37:33.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.908+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d700a1e00 con 0x7f5d700d1470 2024-03-23T14:37:33.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.908+0000 7f5d677fe640 1 -- 172.21.15.161:0/3021491842 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5d60004cf0 con 0x7f5d700d1470 2024-03-23T14:37:33.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.909+0000 7f5d677fe640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d5006b810 0x7f5d5006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:33.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.909+0000 7f5d677fe640 1 -- 172.21.15.161:0/3021491842 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7f5d600896e0 con 0x7f5d700d1470 2024-03-23T14:37:33.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.909+0000 7f5d67fff640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d5006b810 0x7f5d5006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:33.905 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.909+0000 7f5d67fff640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d5006b810 0x7f5d5006dcd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5d58036000 tx=0x7f5d58030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:33.908 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:33.913+0000 7f5d677fe640 1 -- 172.21.15.161:0/3021491842 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5d60054190 con 0x7f5d700d1470 2024-03-23T14:37:34.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.132+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d7004f550 con 0x7f5d700d1470 2024-03-23T14:37:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.133+0000 7f5d677fe640 1 -- 172.21.15.161:0/3021491842 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5193 (secure 0 0 0) 0x7f5d60057850 con 0x7f5d700d1470 2024-03-23T14:37:34.129 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:34.129 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2496234294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":2496234294},{"type":"v1","addr":"172.21.15.183:6836","nonce":2496234294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5396,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2645821131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2645821131},{"type":"v1","addr":"172.21.15.183:6837","nonce":2645821131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/2183619206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2183619206},{"type":"v1","addr":"172.21.15.161:6834","nonce":2183619206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:58.503230+0000","modified":"2024-03-23T14:37:00.601296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/1671955402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":1671955402},{"type":"v1","addr":"172.21.15.161:6835","nonce":1671955402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5925,"qdb_cluster":[5925]},"id":3}]} 2024-03-23T14:37:34.129 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-23T14:37:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d5006b810 msgr2=0x7f5d5006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d5006b810 0x7f5d5006dcd0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f5d58036000 tx=0x7f5d58030000 comp rx=0 tx=0).stop 2024-03-23T14:37:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 msgr2=0x7f5d70158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d70158f80 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f5d6000c500 tx=0x7f5d6000c9c0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 shutdown_connections 2024-03-23T14:37:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5d5006b810 0x7f5d5006dcd0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5d7015e400 0x7f5d700b4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5d700d2970 0x7f5d700b81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 --2- 172.21.15.161:0/3021491842 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5d700d1470 0x7f5d70158f80 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 >> 172.21.15.161:0/3021491842 conn(0x7f5d7009d190 msgr2=0x7f5d70161a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 shutdown_connections 2024-03-23T14:37:34.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.135+0000 7f5d756ed640 1 -- 172.21.15.161:0/3021491842 wait complete. 2024-03-23T14:37:34.142 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:37:34.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.249+0000 7eff0529d640 1 Processor -- start 2024-03-23T14:37:34.245 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.249+0000 7eff0529d640 1 -- start start 2024-03-23T14:37:34.246 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7eff0529d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff0006eb30 0x7eff000b4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7eff0529d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7eff000b4260 0x7eff0015b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7eff0529d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff00070b90 con 0x7eff000b3610 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7eff0529d640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7eff0015ab90 con 0x7eff000b4260 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7eff0529d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7eff000713a0 con 0x7eff0006eb30 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efef67fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7eff000b4260 0x7eff0015b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efef67fc640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7eff000b4260 0x7eff0015b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60598/0 (socket says 172.21.15.161:60598) 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efef67fc640 1 -- 172.21.15.161:0/3757369770 learned_addr learned my addr 172.21.15.161:0/3757369770 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efefeffd640 1 --2- 172.21.15.161:0/3757369770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff0006eb30 0x7eff000b4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efefe7fc640 1 -- 172.21.15.161:0/3757369770 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3833207079 0 0) 0x7eff00070b90 con 0x7eff000b3610 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efefe7fc640 1 -- 172.21.15.161:0/3757369770 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efee0003520 con 0x7eff000b3610 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efefeffd640 1 -- 172.21.15.161:0/3757369770 >> v2:172.21.15.161:3300/0 conn(0x7eff000b4260 msgr2=0x7eff0015b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:34.247 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efefeffd640 1 --2- 172.21.15.161:0/3757369770 >> v2:172.21.15.161:3300/0 conn(0x7eff000b4260 0x7eff0015b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efefeffd640 1 -- 172.21.15.161:0/3757369770 >> v1:172.21.15.161:6789/0 conn(0x7eff000b3610 legacy=0x7eff0006d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.250+0000 7efefeffd640 1 -- 172.21.15.161:0/3757369770 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff0015b0d0 con 0x7eff0006eb30 2024-03-23T14:37:34.248 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.252+0000 7efefeffd640 1 --2- 172.21.15.161:0/3757369770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff0006eb30 0x7eff000b4870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efee80097a0 tx=0x7efee802b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=623915cb1aa8daad server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.253+0000 7efefe7fc640 1 -- 172.21.15.161:0/3757369770 <== mon.2 v2:172.21.15.183:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee802d040 con 0x7eff0006eb30 2024-03-23T14:37:34.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.253+0000 7efefe7fc640 1 -- 172.21.15.161:0/3757369770 <== mon.2 v2:172.21.15.183:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee8031810 con 0x7eff0006eb30 2024-03-23T14:37:34.249 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.253+0000 7efefe7fc640 1 -- 172.21.15.161:0/3757369770 <== mon.2 v2:172.21.15.183:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee8031ba0 con 0x7eff0006eb30 2024-03-23T14:37:34.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.254+0000 7eff0529d640 1 -- 172.21.15.161:0/3757369770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff0006eb30 msgr2=0x7eff000b4870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.254+0000 7eff0529d640 1 --2- 172.21.15.161:0/3757369770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff0006eb30 0x7eff000b4870 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7efee80097a0 tx=0x7efee802b8b0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.254+0000 7eff0529d640 1 -- 172.21.15.161:0/3757369770 shutdown_connections 2024-03-23T14:37:34.250 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.254+0000 7eff0529d640 1 --2- 172.21.15.161:0/3757369770 >> v2:172.21.15.161:3300/0 conn(0x7eff000b4260 0x7eff0015b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.254+0000 7eff0529d640 1 --2- 172.21.15.161:0/3757369770 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff0006eb30 0x7eff000b4870 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.254+0000 7eff0529d640 1 -- 172.21.15.161:0/3757369770 >> 172.21.15.161:0/3757369770 conn(0x7eff000b17f0 msgr2=0x7eff000b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.254+0000 7eff0529d640 1 -- 172.21.15.161:0/3757369770 shutdown_connections 2024-03-23T14:37:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.254+0000 7eff0529d640 1 -- 172.21.15.161:0/3757369770 wait complete. 2024-03-23T14:37:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.255+0000 7eff0529d640 1 Processor -- start 2024-03-23T14:37:34.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.255+0000 7eff0529d640 1 -- start start 2024-03-23T14:37:34.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7eff0529d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff0006eb30 0x7eff000ce170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7eff0529d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff000b3610 0x7eff00154030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7eff0529d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff000b4260 0x7eff00154570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7eff0529d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7eff000cdfc0 con 0x7eff000b3610 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7eff0529d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7eff000cccd0 con 0x7eff0006eb30 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7eff0529d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7eff0015dc20 con 0x7eff000b4260 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7efef67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff000b3610 0x7eff00154030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7efef67fc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff000b3610 0x7eff00154030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60606/0 (socket says 172.21.15.161:60606) 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7efef67fc640 1 -- 172.21.15.161:0/2226424641 learned_addr learned my addr 172.21.15.161:0/2226424641 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7efeff7fe640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff000b4260 0x7eff00154570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.256+0000 7efefeffd640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff0006eb30 0x7eff000ce170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef67fc640 1 -- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff000b4260 msgr2=0x7eff00154570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef67fc640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff000b4260 0x7eff00154570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef67fc640 1 -- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff0006eb30 msgr2=0x7eff000ce170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef67fc640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff0006eb30 0x7eff000ce170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef67fc640 1 -- 172.21.15.161:0/2226424641 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efee8009400 con 0x7eff000b3610 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef67fc640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff000b3610 0x7eff00154030 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efef000e870 tx=0x7efef000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef7fff640 1 -- 172.21.15.161:0/2226424641 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef000c8e0 con 0x7eff000b3610 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef7fff640 1 -- 172.21.15.161:0/2226424641 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efef000ca40 con 0x7eff000b3610 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff000ccf20 con 0x7eff000b3610 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7efef7fff640 1 -- 172.21.15.161:0/2226424641 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efef000cdf0 con 0x7eff000b3610 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.257+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff000cd4e0 con 0x7eff000b3610 2024-03-23T14:37:34.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.258+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff000ccab0 con 0x7eff000b3610 2024-03-23T14:37:34.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.259+0000 7efef7fff640 1 -- 172.21.15.161:0/2226424641 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efef0013470 con 0x7eff000b3610 2024-03-23T14:37:34.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.259+0000 7efef7fff640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efee406b810 0x7efee406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.259+0000 7efef7fff640 1 -- 172.21.15.161:0/2226424641 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7efef008a1e0 con 0x7eff000b3610 2024-03-23T14:37:34.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.259+0000 7efefeffd640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efee406b810 0x7efee406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.260+0000 7efefeffd640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efee406b810 0x7efee406dcd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efee8036000 tx=0x7efee803c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.264+0000 7efef7fff640 1 -- 172.21.15.161:0/2226424641 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efef0054d70 con 0x7eff000b3610 2024-03-23T14:37:34.475 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.480+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7eff0004f550 con 0x7eff000b3610 2024-03-23T14:37:34.476 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.480+0000 7efef7fff640 1 -- 172.21.15.161:0/2226424641 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+5193 (secure 0 0 0) 0x7efef0058430 con 0x7eff000b3610 2024-03-23T14:37:34.476 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:34.476 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":23,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2496234294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":2496234294},{"type":"v1","addr":"172.21.15.183:6836","nonce":2496234294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5396,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2645821131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2645821131},{"type":"v1","addr":"172.21.15.183:6837","nonce":2645821131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/2183619206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2183619206},{"type":"v1","addr":"172.21.15.161:6834","nonce":2183619206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19}],"filesystems":[{"mdsmap":{"epoch":23,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-23T14:36:58.503230+0000","modified":"2024-03-23T14:37:00.601296+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":5925},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5925":{"gid":5925,"name":"a","rank":0,"incarnation":21,"state":"up:active","state_seq":3,"addr":"172.21.15.161:6835/1671955402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":1671955402},{"type":"v1","addr":"172.21.15.161:6835","nonce":1671955402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":5925,"qdb_cluster":[5925]},"id":3}]} 2024-03-23T14:37:34.477 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 23 2024-03-23T14:37:34.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efee406b810 msgr2=0x7efee406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efee406b810 0x7efee406dcd0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7efee8036000 tx=0x7efee803c040 comp rx=0 tx=0).stop 2024-03-23T14:37:34.478 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff000b3610 msgr2=0x7eff00154030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff000b3610 0x7eff00154030 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7efef000e870 tx=0x7efef000ed30 comp rx=0 tx=0).stop 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 shutdown_connections 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efee406b810 0x7efee406dcd0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7eff000b4260 0x7eff00154570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7eff000b3610 0x7eff00154030 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 --2- 172.21.15.161:0/2226424641 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7eff0006eb30 0x7eff000ce170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 >> 172.21.15.161:0/2226424641 conn(0x7eff000b17f0 msgr2=0x7eff0015e110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 shutdown_connections 2024-03-23T14:37:34.479 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.482+0000 7eff0529d640 1 -- 172.21.15.161:0/2226424641 wait complete. 2024-03-23T14:37:34.489 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:34.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.595+0000 7fb5ad0c3640 1 Processor -- start 2024-03-23T14:37:34.591 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.595+0000 7fb5ad0c3640 1 -- start start 2024-03-23T14:37:34.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5ad0c3640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5a815e3b0 0x7fb5a80d2460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5ad0c3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb5a80b2ee0 0x7fb5a80d29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5ad0c3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5a80b34f0 con 0x7fb5a815e790 2024-03-23T14:37:34.593 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5ad0c3640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb5a80d0420 con 0x7fb5a815e3b0 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5ad0c3640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fb5a80d4720 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a6d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5a815e3b0 0x7fb5a80d2460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a6d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb5a815e3b0 0x7fb5a80d2460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60618/0 (socket says 172.21.15.161:60618) 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a6d9d640 1 -- 172.21.15.161:0/2810659513 learned_addr learned my addr 172.21.15.161:0/2810659513 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a759e640 1 --2- 172.21.15.161:0/2810659513 >> v2:172.21.15.183:3300/0 conn(0x7fb5a80b2ee0 0x7fb5a80d29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a6d9d640 1 -- 172.21.15.161:0/2810659513 >> v2:172.21.15.183:3300/0 conn(0x7fb5a80b2ee0 msgr2=0x7fb5a80d29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a6d9d640 1 --2- 172.21.15.161:0/2810659513 >> v2:172.21.15.183:3300/0 conn(0x7fb5a80b2ee0 0x7fb5a80d29a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a6d9d640 1 -- 172.21.15.161:0/2810659513 >> v1:172.21.15.183:6789/0 conn(0x7fb5a815e790 legacy=0x7fb5a80b39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a6d9d640 1 -- 172.21.15.161:0/2810659513 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5a80d4a00 con 0x7fb5a815e3b0 2024-03-23T14:37:34.594 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.597+0000 7fb5a759e640 1 --2- 172.21.15.161:0/2810659513 >> v2:172.21.15.183:3300/0 conn(0x7fb5a80b2ee0 0x7fb5a80d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:34.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.598+0000 7fb5a6d9d640 1 --2- 172.21.15.161:0/2810659513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a815e3b0 0x7fb5a80d2460 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb5a808c9c0 tx=0x7fb59802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc73b1a22d5b5590 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.595 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.598+0000 7fb5a5d9b640 1 -- 172.21.15.161:0/2810659513 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598031870 con 0x7fb5a815e3b0 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.598+0000 7fb5a5d9b640 1 -- 172.21.15.161:0/2810659513 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5980319d0 con 0x7fb5a815e3b0 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.598+0000 7fb5a5d9b640 1 -- 172.21.15.161:0/2810659513 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598031d60 con 0x7fb5a815e3b0 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.599+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/2810659513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a815e3b0 msgr2=0x7fb5a80d2460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.599+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/2810659513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a815e3b0 0x7fb5a80d2460 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7fb5a808c9c0 tx=0x7fb59802b910 comp rx=0 tx=0).stop 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.599+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/2810659513 shutdown_connections 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.599+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/2810659513 >> v2:172.21.15.183:3300/0 conn(0x7fb5a80b2ee0 0x7fb5a80d29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.599+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/2810659513 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a815e3b0 0x7fb5a80d2460 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.599+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/2810659513 >> 172.21.15.161:0/2810659513 conn(0x7fb5a809d1a0 msgr2=0x7fb5a80d3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.599+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/2810659513 shutdown_connections 2024-03-23T14:37:34.596 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.599+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/2810659513 wait complete. 2024-03-23T14:37:34.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.600+0000 7fb5ad0c3640 1 Processor -- start 2024-03-23T14:37:34.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5ad0c3640 1 -- start start 2024-03-23T14:37:34.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5ad0c3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a80b2ee0 0x7fb5a80b94c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5ad0c3640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a815e3b0 0x7fb5a806bd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5ad0c3640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a815e790 0x7fb5a80c79a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.598 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5ad0c3640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a80cbe10 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5ad0c3640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb5a80cc0f0 con 0x7fb5a815e790 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5ad0c3640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb5a80cc3d0 con 0x7fb5a815e3b0 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5a6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a80b2ee0 0x7fb5a80b94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5a6d9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a80b2ee0 0x7fb5a80b94c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60632/0 (socket says 172.21.15.161:60632) 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.601+0000 7fb5a6d9d640 1 -- 172.21.15.161:0/3799687060 learned_addr learned my addr 172.21.15.161:0/3799687060 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb5a6d9d640 1 -- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a815e3b0 msgr2=0x7fb5a806bd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb5a6d9d640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a815e3b0 0x7fb5a806bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb5a6d9d640 1 -- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a815e790 msgr2=0x7fb5a80c79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb5a6d9d640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a815e790 0x7fb5a80c79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb5a6d9d640 1 -- 172.21.15.161:0/3799687060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5980094d0 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb5a6d9d640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a80b2ee0 0x7fb5a80b94c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb598000c00 tx=0x7fb5980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb58f7fe640 1 -- 172.21.15.161:0/3799687060 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb598031840 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb58f7fe640 1 -- 172.21.15.161:0/3799687060 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5980443f0 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.599 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5a80cc6b0 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb58f7fe640 1 -- 172.21.15.161:0/3799687060 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5980447a0 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.600 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.602+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5a80b9a00 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.603+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb5a80ccbd0 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.603+0000 7fb58f7fe640 1 -- 172.21.15.161:0/3799687060 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb59804d450 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.604+0000 7fb58f7fe640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb59406b810 0x7fb59406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.604+0000 7fb58f7fe640 1 -- 172.21.15.161:0/3799687060 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fb59802d040 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.604+0000 7fb5a659c640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb59406b810 0x7fb59406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.601 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.605+0000 7fb5a659c640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb59406b810 0x7fb59406dcd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb59c006fd0 tx=0x7fb59c006bc0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.604 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.608+0000 7fb58f7fe640 1 -- 172.21.15.161:0/3799687060 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb5980773f0 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.830 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.834+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb5a80a1e80 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.831 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.835+0000 7fb58f7fe640 1 -- 172.21.15.161:0/3799687060 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v48) ==== 74+0+16705 (secure 0 0 0) 0x7fb598035090 con 0x7fb5a80b2ee0 2024-03-23T14:37:34.831 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:34.831 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":48,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:00.168152+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2024-03-23T14:36:56.747667+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":8,"pool_name":"cephfs_data","create_time":"2024-03-23T14:36:57.478793+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb59406b810 msgr2=0x7fb59406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb59406b810 0x7fb59406dcd0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fb59c006fd0 tx=0x7fb59c006bc0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a80b2ee0 msgr2=0x7fb5a80b94c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a80b2ee0 0x7fb5a80b94c0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7fb598000c00 tx=0x7fb5980046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 shutdown_connections 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb59406b810 0x7fb59406dcd0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb5a815e790 0x7fb5a80c79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb5a815e3b0 0x7fb5a806bd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 --2- 172.21.15.161:0/3799687060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb5a80b2ee0 0x7fb5a80b94c0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 >> 172.21.15.161:0/3799687060 conn(0x7fb5a809d1a0 msgr2=0x7fb5a80b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:34.833 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 shutdown_connections 2024-03-23T14:37:34.834 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.837+0000 7fb5ad0c3640 1 -- 172.21.15.161:0/3799687060 wait complete. 2024-03-23T14:37:34.845 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-23T14:37:34.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.951+0000 7fc11e3e2640 1 Processor -- start 2024-03-23T14:37:34.947 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.951+0000 7fc11e3e2640 1 -- start start 2024-03-23T14:37:34.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc11e3e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc1180d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.948 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc11e3e2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc1180d2970 0x7fc1180b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc11e3e2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1180d2dd0 con 0x7fc11815e400 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc11e3e2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fc1180d1050 con 0x7fc1180d2970 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc11e3e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc1180d0460 con 0x7fc1180d1470 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc1180d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc11c97f640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fc1180d2970 0x7fc1180b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc117fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fc11815e400 0x7fc1180cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:52264/0 (socket says 172.21.15.161:52264) 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc117fff640 1 -- 172.21.15.161:0/2130251397 learned_addr learned my addr 172.21.15.161:0/2130251397 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.952+0000 7fc1177fe640 1 --2- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc1180d2430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:35980/0 (socket says 172.21.15.161:35980) 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.953+0000 7fc11c97f640 1 -- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 msgr2=0x7fc1180d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.953+0000 7fc11c97f640 1 --2- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc1180d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.953+0000 7fc11c97f640 1 -- 172.21.15.161:0/2130251397 >> v1:172.21.15.183:6789/0 conn(0x7fc11815e400 legacy=0x7fc1180cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.953+0000 7fc116ffd640 1 -- 172.21.15.161:0/2130251397 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1198458295 0 0) 0x7fc1180d2dd0 con 0x7fc11815e400 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.953+0000 7fc11c97f640 1 -- 172.21.15.161:0/2130251397 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1180d06b0 con 0x7fc1180d2970 2024-03-23T14:37:34.949 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.953+0000 7fc1177fe640 1 --2- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc1180d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.954+0000 7fc11c97f640 1 --2- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1180d2970 0x7fc1180b0a80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc110009d90 tx=0x7fc11002bea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc34e0aa7644059f server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.954+0000 7fc116ffd640 1 -- 172.21.15.161:0/2130251397 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc11002cc50 con 0x7fc1180d2970 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.954+0000 7fc116ffd640 1 -- 172.21.15.161:0/2130251397 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc11002cdb0 con 0x7fc1180d2970 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.954+0000 7fc116ffd640 1 -- 172.21.15.161:0/2130251397 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc110034660 con 0x7fc1180d2970 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.955+0000 7fc11e3e2640 1 -- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1180d2970 msgr2=0x7fc1180b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.955+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1180d2970 0x7fc1180b0a80 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fc110009d90 tx=0x7fc11002bea0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.955+0000 7fc11e3e2640 1 -- 172.21.15.161:0/2130251397 shutdown_connections 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.955+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc1180d2970 0x7fc1180b0a80 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.955+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/2130251397 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc1180d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.955+0000 7fc11e3e2640 1 -- 172.21.15.161:0/2130251397 >> 172.21.15.161:0/2130251397 conn(0x7fc11809d190 msgr2=0x7fc1180d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:34.951 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.955+0000 7fc11e3e2640 1 -- 172.21.15.161:0/2130251397 shutdown_connections 2024-03-23T14:37:34.952 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.955+0000 7fc11e3e2640 1 -- 172.21.15.161:0/2130251397 wait complete. 2024-03-23T14:37:34.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.956+0000 7fc11e3e2640 1 Processor -- start 2024-03-23T14:37:34.953 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.956+0000 7fc11e3e2640 1 -- start start 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11e3e2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc118159f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11e3e2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1180d2970 0x7fc1180b8ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11e3e2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc11815e400 0x7fc1180b9400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11e3e2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fc1181f5100 con 0x7fc1180d2970 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11e3e2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fc11815a4d0 con 0x7fc11815e400 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11e3e2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fc1181f44f0 con 0x7fc1180d1470 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1180d2970 0x7fc1180b8ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1180d2970 0x7fc1180b8ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60638/0 (socket says 172.21.15.161:60638) 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 -- 172.21.15.161:0/4107415828 learned_addr learned my addr 172.21.15.161:0/4107415828 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc117fff640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc118159f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11c97f640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc11815e400 0x7fc1180b9400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 -- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 msgr2=0x7fc118159f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.954 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc118159f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 -- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc11815e400 msgr2=0x7fc1180b9400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc11815e400 0x7fc1180b9400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 -- 172.21.15.161:0/4107415828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c009400 con 0x7fc1180d2970 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc1177fe640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1180d2970 0x7fc1180b8ec0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00e960 tx=0x7fc10c00ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc114ff9640 1 -- 172.21.15.161:0/4107415828 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c00ca00 con 0x7fc1180d2970 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc114ff9640 1 -- 172.21.15.161:0/4107415828 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc10c00cb60 con 0x7fc1180d2970 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc114ff9640 1 -- 172.21.15.161:0/4107415828 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc10c013470 con 0x7fc1180d2970 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc110009a70 con 0x7fc1180d2970 2024-03-23T14:37:34.955 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.957+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1181f4ab0 con 0x7fc1180d2970 2024-03-23T14:37:34.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.959+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1180a1e00 con 0x7fc1180d2970 2024-03-23T14:37:34.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.959+0000 7fc114ff9640 1 -- 172.21.15.161:0/4107415828 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fc10c0135d0 con 0x7fc1180d2970 2024-03-23T14:37:34.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.959+0000 7fc114ff9640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc10406b810 0x7fc10406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:34.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.959+0000 7fc114ff9640 1 -- 172.21.15.161:0/4107415828 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(48..48 src has 1..48) ==== 5709+0+0 (secure 0 0 0) 0x7fc10c08a880 con 0x7fc1180d2970 2024-03-23T14:37:34.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.960+0000 7fc117fff640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc10406b810 0x7fc10406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:34.956 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.960+0000 7fc117fff640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc10406b810 0x7fc10406dcd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc108000f60 tx=0x7fc108009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:34.959 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:34.964+0000 7fc114ff9640 1 -- 172.21.15.161:0/4107415828 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fc10c055210 con 0x7fc1180d2970 2024-03-23T14:37:35.175 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.180+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc1180b9c60 con 0x7fc1180d2970 2024-03-23T14:37:35.176 INFO:tasks.ceph.mon.a.smithi161.stderr:2024-03-23T14:37:35.180+0000 7f8d40811640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-23T14:37:35.177 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.182+0000 7fc114ff9640 1 -- 172.21.15.161:0/4107415828 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v24) ==== 161+0+0 (secure 0 0 0) 0x7fc10c017090 con 0x7fc1180d2970 2024-03-23T14:37:35.177 INFO:teuthology.orchestra.run.smithi161.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc10406b810 msgr2=0x7fc10406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc10406b810 0x7fc10406dcd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc108000f60 tx=0x7fc108009040 comp rx=0 tx=0).stop 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1180d2970 msgr2=0x7fc1180b8ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1180d2970 0x7fc1180b8ec0 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fc10c00e960 tx=0x7fc10c00ee20 comp rx=0 tx=0).stop 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 shutdown_connections 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fc10406b810 0x7fc10406dcd0 secure :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc108000f60 tx=0x7fc108009040 comp rx=0 tx=0).stop 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fc11815e400 0x7fc1180b9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fc1180d2970 0x7fc1180b8ec0 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 --2- 172.21.15.161:0/4107415828 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fc1180d1470 0x7fc118159f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.185+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 >> 172.21.15.161:0/4107415828 conn(0x7fc11809d190 msgr2=0x7fc1180b2070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:35.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.186+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 shutdown_connections 2024-03-23T14:37:35.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.186+0000 7fc11e3e2640 1 -- 172.21.15.161:0/4107415828 wait complete. 2024-03-23T14:37:35.183 INFO:tasks.ceph.mds.a.smithi161.stderr: -2918> 2024-03-23T14:37:04.439+0000 7f76d14c5640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-23T14:37:35.193 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-23T14:37:35.211 INFO:tasks.ceph.mds.a.smithi161.stdout:starting mds.a at 2024-03-23T14:37:35.297 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.302+0000 7f745bee6640 1 Processor -- start 2024-03-23T14:37:35.298 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.302+0000 7f745bee6640 1 -- start start 2024-03-23T14:37:35.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745bee6640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f74540b4010 0x7f74540cfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:35.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745bee6640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f745415bb90 0x7f74540d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:35.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745bee6640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f745415c1f0 con 0x7f74540b43f0 2024-03-23T14:37:35.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745bee6640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f745406ef20 con 0x7f745415bb90 2024-03-23T14:37:35.299 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745bee6640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f745415f730 con 0x7f74540b4010 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745a483640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f745415bb90 0x7f74540d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745a483640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f745415bb90 0x7f74540d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60656/0 (socket says 172.21.15.161:60656) 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745a483640 1 -- 172.21.15.161:0/1474176435 learned_addr learned my addr 172.21.15.161:0/1474176435 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f7459c82640 1 --2- 172.21.15.161:0/1474176435 >> v2:172.21.15.183:3300/0 conn(0x7f74540b4010 0x7f74540cfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745a483640 1 -- 172.21.15.161:0/1474176435 >> v2:172.21.15.183:3300/0 conn(0x7f74540b4010 msgr2=0x7f74540cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745a483640 1 --2- 172.21.15.161:0/1474176435 >> v2:172.21.15.183:3300/0 conn(0x7f74540b4010 0x7f74540cfee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745a483640 1 -- 172.21.15.161:0/1474176435 >> v1:172.21.15.161:6789/0 conn(0x7f74540b43f0 legacy=0x7f745415b480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f745a483640 1 -- 172.21.15.161:0/1474176435 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f745415e970 con 0x7f745415bb90 2024-03-23T14:37:35.300 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.303+0000 7f7458c80640 1 -- 172.21.15.161:0/1474176435 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4262587725 0 0) 0x7f745415c1f0 con 0x7f74540b43f0 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.304+0000 7f745a483640 1 --2- 172.21.15.161:0/1474176435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745415bb90 0x7f74540d0420 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f744c009e50 tx=0x7f744c02bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493fe4be5e86ec4d server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.304+0000 7f7458c80640 1 -- 172.21.15.161:0/1474176435 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c02ca80 con 0x7f745415bb90 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.304+0000 7f7458c80640 1 -- 172.21.15.161:0/1474176435 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f744c02cbe0 con 0x7f745415bb90 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.304+0000 7f7458c80640 1 -- 172.21.15.161:0/1474176435 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f744c0343f0 con 0x7f745415bb90 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.304+0000 7f745bee6640 1 -- 172.21.15.161:0/1474176435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745415bb90 msgr2=0x7f74540d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.305+0000 7f745bee6640 1 --2- 172.21.15.161:0/1474176435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745415bb90 0x7f74540d0420 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f744c009e50 tx=0x7f744c02bee0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.305+0000 7f745bee6640 1 -- 172.21.15.161:0/1474176435 shutdown_connections 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.305+0000 7f745bee6640 1 --2- 172.21.15.161:0/1474176435 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f745415bb90 0x7f74540d0420 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.302 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.305+0000 7f745bee6640 1 --2- 172.21.15.161:0/1474176435 >> v2:172.21.15.183:3300/0 conn(0x7f74540b4010 0x7f74540cfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.305+0000 7f745bee6640 1 -- 172.21.15.161:0/1474176435 >> 172.21.15.161:0/1474176435 conn(0x7f745406b4a0 msgr2=0x7f745406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:35.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.305+0000 7f745bee6640 1 -- 172.21.15.161:0/1474176435 shutdown_connections 2024-03-23T14:37:35.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.305+0000 7f745bee6640 1 -- 172.21.15.161:0/1474176435 wait complete. 2024-03-23T14:37:35.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.306+0000 7f745bee6640 1 Processor -- start 2024-03-23T14:37:35.303 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.306+0000 7f745bee6640 1 -- start start 2024-03-23T14:37:35.305 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f745bee6640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74540b4010 0x7f74541532b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f745bee6640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74540b43f0 0x7f74541564d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f745bee6640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f745415bb90 0x7f7454142ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f745bee6640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f74541537f0 con 0x7f74540b4010 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f745bee6640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f745415dce0 con 0x7f745415bb90 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f745bee6640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f74541524f0 con 0x7f74540b43f0 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74540b4010 0x7f74541532b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74540b4010 0x7f74541532b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60672/0 (socket says 172.21.15.161:60672) 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 -- 172.21.15.161:0/2925890312 learned_addr learned my addr 172.21.15.161:0/2925890312 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 -- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74540b43f0 msgr2=0x7f74541564d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74540b43f0 0x7f74541564d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.306 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 -- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f745415bb90 msgr2=0x7f7454142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f745415bb90 0x7f7454142ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 -- 172.21.15.161:0/2925890312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f744c009b30 con 0x7f74540b4010 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f7459c82640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74540b4010 0x7f74541532b0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f745000fa70 tx=0x7f745000ff30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f744a7fc640 1 -- 172.21.15.161:0/2925890312 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7450007c90 con 0x7f74540b4010 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f744a7fc640 1 -- 172.21.15.161:0/2925890312 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7450007df0 con 0x7f74540b4010 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74541527d0 con 0x7f74540b4010 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f744a7fc640 1 -- 172.21.15.161:0/2925890312 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f745001a680 con 0x7f74540b4010 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.307+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7454152d90 con 0x7f74540b4010 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.308+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f745406de90 con 0x7f74540b4010 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.309+0000 7f744a7fc640 1 -- 172.21.15.161:0/2925890312 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f745001a7e0 con 0x7f74540b4010 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.309+0000 7f744a7fc640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f743806b810 0x7f743806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:35.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.309+0000 7f744a7fc640 1 -- 172.21.15.161:0/2925890312 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5795+0+0 (secure 0 0 0) 0x7f745008ca90 con 0x7f74540b4010 2024-03-23T14:37:35.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.309+0000 7f7459481640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f743806b810 0x7f743806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:35.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.310+0000 7f7459481640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f743806b810 0x7f743806dcd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f745415e710 tx=0x7f7444007910 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:35.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.313+0000 7f744a7fc640 1 -- 172.21.15.161:0/2925890312 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7450057450 con 0x7f74540b4010 2024-03-23T14:37:35.517 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:35.521+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f7454073570 con 0x7f74540b4010 2024-03-23T14:37:36.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.183+0000 7f744a7fc640 1 -- 172.21.15.161:0/2925890312 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v25) ==== 104+0+0 (secure 0 0 0) 0x7f745005ab10 con 0x7f74540b4010 2024-03-23T14:37:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f743806b810 msgr2=0x7f743806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f743806b810 0x7f743806dcd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f745415e710 tx=0x7f7444007910 comp rx=0 tx=0).stop 2024-03-23T14:37:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74540b4010 msgr2=0x7f74541532b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74540b4010 0x7f74541532b0 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f745000fa70 tx=0x7f745000ff30 comp rx=0 tx=0).stop 2024-03-23T14:37:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 shutdown_connections 2024-03-23T14:37:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f743806b810 0x7f743806dcd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f745415bb90 0x7f7454142ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f74540b43f0 0x7f74541564d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 --2- 172.21.15.161:0/2925890312 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f74540b4010 0x7f74541532b0 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 >> 172.21.15.161:0/2925890312 conn(0x7f745406b4a0 msgr2=0x7f745415b0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:36.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 shutdown_connections 2024-03-23T14:37:36.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.186+0000 7f745bee6640 1 -- 172.21.15.161:0/2925890312 wait complete. 2024-03-23T14:37:36.195 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-23T14:37:36.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.348+0000 7fe4d26fb640 1 Processor -- start 2024-03-23T14:37:36.344 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.348+0000 7fe4d26fb640 1 -- start start 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4d26fb640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe4cc145720 0x7fe4cc160310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4d26fb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe4cc0cfde0 0x7fe4cc15c350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4d26fb640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe4cc145b00 con 0x7fe4cc15c890 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4d26fb640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe4cc160850 con 0x7fe4cc0cfde0 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4d26fb640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fe4cc15cdd0 con 0x7fe4cc145720 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4cb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe4cc0cfde0 0x7fe4cc15c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4d0c98640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fe4cc15c890 0x7fe4cc0ce790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42852/0 (socket says 172.21.15.161:42852) 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4cb7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe4cc0cfde0 0x7fe4cc15c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60678/0 (socket says 172.21.15.161:60678) 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4d0c98640 1 -- 172.21.15.161:0/2770997153 learned_addr learned my addr 172.21.15.161:0/2770997153 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4cbfff640 1 --2- 172.21.15.161:0/2770997153 >> v2:172.21.15.183:3300/0 conn(0x7fe4cc145720 0x7fe4cc160310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4cb7fe640 1 -- 172.21.15.161:0/2770997153 >> v2:172.21.15.183:3300/0 conn(0x7fe4cc145720 msgr2=0x7fe4cc160310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4cb7fe640 1 --2- 172.21.15.161:0/2770997153 >> v2:172.21.15.183:3300/0 conn(0x7fe4cc145720 0x7fe4cc160310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.347 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4cb7fe640 1 -- 172.21.15.161:0/2770997153 >> v1:172.21.15.161:6789/0 conn(0x7fe4cc15c890 legacy=0x7fe4cc0ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4cb7fe640 1 -- 172.21.15.161:0/2770997153 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4cc15d390 con 0x7fe4cc0cfde0 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4caffd640 1 -- 172.21.15.161:0/2770997153 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1170271520 0 0) 0x7fe4cc145b00 con 0x7fe4cc15c890 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.349+0000 7fe4cbfff640 1 --2- 172.21.15.161:0/2770997153 >> v2:172.21.15.183:3300/0 conn(0x7fe4cc145720 0x7fe4cc160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.350+0000 7fe4cb7fe640 1 --2- 172.21.15.161:0/2770997153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc0cfde0 0x7fe4cc15c350 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0d0b80 tx=0x7fe4c002b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e4ac61f8501d184 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.350+0000 7fe4caffd640 1 -- 172.21.15.161:0/2770997153 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0031830 con 0x7fe4cc0cfde0 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.350+0000 7fe4caffd640 1 -- 172.21.15.161:0/2770997153 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c0031990 con 0x7fe4cc0cfde0 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.350+0000 7fe4caffd640 1 -- 172.21.15.161:0/2770997153 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c0031d20 con 0x7fe4cc0cfde0 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.351+0000 7fe4d26fb640 1 -- 172.21.15.161:0/2770997153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc0cfde0 msgr2=0x7fe4cc15c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.351+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/2770997153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc0cfde0 0x7fe4cc15c350 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc0d0b80 tx=0x7fe4c002b8d0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.351+0000 7fe4d26fb640 1 -- 172.21.15.161:0/2770997153 shutdown_connections 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.351+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/2770997153 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc0cfde0 0x7fe4cc15c350 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.351+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/2770997153 >> v2:172.21.15.183:3300/0 conn(0x7fe4cc145720 0x7fe4cc160310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.351+0000 7fe4d26fb640 1 -- 172.21.15.161:0/2770997153 >> 172.21.15.161:0/2770997153 conn(0x7fe4cc06b260 msgr2=0x7fe4cc06b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:36.348 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.351+0000 7fe4d26fb640 1 -- 172.21.15.161:0/2770997153 shutdown_connections 2024-03-23T14:37:36.349 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.351+0000 7fe4d26fb640 1 -- 172.21.15.161:0/2770997153 wait complete. 2024-03-23T14:37:36.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.352+0000 7fe4d26fb640 1 Processor -- start 2024-03-23T14:37:36.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.352+0000 7fe4d26fb640 1 -- start start 2024-03-23T14:37:36.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d26fb640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4cc0cfde0 0x7fe4cc12fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:36.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d26fb640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4cc145720 0x7fe4cc0a9f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:36.351 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d26fb640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc15c890 0x7fe4cc15b540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d26fb640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe4cc1301a0 con 0x7fe4cc15c890 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d26fb640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe4cc130480 con 0x7fe4cc145720 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d26fb640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe4cc130760 con 0x7fe4cc0cfde0 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc15c890 0x7fe4cc15b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc15c890 0x7fe4cc15b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60686/0 (socket says 172.21.15.161:60686) 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 -- 172.21.15.161:0/1131339307 learned_addr learned my addr 172.21.15.161:0/1131339307 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4cb7fe640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4cc145720 0x7fe4cc0a9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4cbfff640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4cc0cfde0 0x7fe4cc12fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 -- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4cc0cfde0 msgr2=0x7fe4cc12fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4cc0cfde0 0x7fe4cc12fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.352 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 -- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4cc145720 msgr2=0x7fe4cc0a9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4cc145720 0x7fe4cc0a9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 -- 172.21.15.161:0/1131339307 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4c0009400 con 0x7fe4cc15c890 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4cbfff640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4cc0cfde0 0x7fe4cc12fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d0c98640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc15c890 0x7fe4cc15b540 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400c780 tx=0x7fe4c400cc40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4c8ff9640 1 -- 172.21.15.161:0/1131339307 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c400ce30 con 0x7fe4cc15c890 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4c8ff9640 1 -- 172.21.15.161:0/1131339307 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe4c400a640 con 0x7fe4cc15c890 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4cc130a40 con 0x7fe4cc15c890 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.353+0000 7fe4c8ff9640 1 -- 172.21.15.161:0/1131339307 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c400aa30 con 0x7fe4cc15c890 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.354+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4cc0aa460 con 0x7fe4cc15c890 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.355+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4cc130f60 con 0x7fe4cc15c890 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.355+0000 7fe4c8ff9640 1 -- 172.21.15.161:0/1131339307 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe4c400ab90 con 0x7fe4cc15c890 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.355+0000 7fe4c8ff9640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4b806b810 0x7fe4b806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:36.353 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.356+0000 7fe4c8ff9640 1 -- 172.21.15.161:0/1131339307 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(49..49 src has 1..49) ==== 5795+0+0 (secure 0 0 0) 0x7fe4c4089930 con 0x7fe4cc15c890 2024-03-23T14:37:36.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.356+0000 7fe4cbfff640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4b806b810 0x7fe4b806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:36.354 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.356+0000 7fe4cbfff640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4b806b810 0x7fe4b806dcd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc006fd0 tx=0x7fe4bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:36.355 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.360+0000 7fe4c8ff9640 1 -- 172.21.15.161:0/1131339307 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fe4cc130f60 con 0x7fe4cc15c890 2024-03-23T14:37:36.527 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:36.532+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fe4cc130f60 con 0x7fe4cc15c890 2024-03-23T14:37:37.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.261+0000 7fe4c8ff9640 1 -- 172.21.15.161:0/1131339307 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v50) ==== 181+0+0 (secure 0 0 0) 0x7fe4cc130f60 con 0x7fe4cc15c890 2024-03-23T14:37:37.257 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_metadata' removed 2024-03-23T14:37:37.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4b806b810 msgr2=0x7fe4b806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4b806b810 0x7fe4b806dcd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc006fd0 tx=0x7fe4bc008040 comp rx=0 tx=0).stop 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc15c890 msgr2=0x7fe4cc15b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc15c890 0x7fe4cc15b540 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400c780 tx=0x7fe4c400cc40 comp rx=0 tx=0).stop 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 shutdown_connections 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe4b806b810 0x7fe4b806dcd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe4cc15c890 0x7fe4cc15b540 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe4cc145720 0x7fe4cc0a9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.265+0000 7fe4d26fb640 1 --2- 172.21.15.161:0/1131339307 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe4cc0cfde0 0x7fe4cc12fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.266+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 >> 172.21.15.161:0/1131339307 conn(0x7fe4cc06b260 msgr2=0x7fe4cc15ef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:37.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.267+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 shutdown_connections 2024-03-23T14:37:37.263 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.267+0000 7fe4d26fb640 1 -- 172.21.15.161:0/1131339307 wait complete. 2024-03-23T14:37:37.277 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-23T14:37:37.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.388+0000 7fbf91f81640 1 Processor -- start 2024-03-23T14:37:37.384 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.388+0000 7fbf91f81640 1 -- start start 2024-03-23T14:37:37.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf91f81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c071860 0x7fbf8c071cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:37.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf91f81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8c144ac0 con 0x7fbf8c1446e0 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf91f81640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf8c15b750 con 0x7fbf8c15bbe0 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf91f81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c072200 con 0x7fbf8c071860 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8b7fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fbf8c1446e0 0x7fbf8c06de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:42866/0 (socket says 172.21.15.161:42866) 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8b7fe640 1 -- 172.21.15.161:0/2322762756 learned_addr learned my addr 172.21.15.161:0/2322762756 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3470308654 0 0) 0x7fbf8c144ac0 con 0x7fbf8c1446e0 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c071860 0x7fbf8c071cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf74003520 con 0x7fbf8c1446e0 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3431811330 0 0) 0x7fbf74003520 con 0x7fbf8c1446e0 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c071860 msgr2=0x7fbf8c071cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c071860 0x7fbf8c071cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 >> v1:172.21.15.183:6789/0 conn(0x7fbf8c15bbe0 legacy=0x7fbf8c15c070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c06e550 con 0x7fbf8c1446e0 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7fbf7c001670 con 0x7fbf8c1446e0 2024-03-23T14:37:37.388 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 >> v1:172.21.15.161:6789/0 conn(0x7fbf8c1446e0 legacy=0x7fbf8c06de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf740039a0 0x7fbf74003d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf74004820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c071860 0x7fbf74005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c072200 con 0x7fbf74004350 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbf7400b980 con 0x7fbf740039a0 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fbf7400dd50 con 0x7fbf8c071860 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.389+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf74004820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8b7fe640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf740039a0 0x7fbf74003d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8bfff640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c071860 0x7fbf74005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8affd640 1 -- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c071860 msgr2=0x7fbf74005080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c071860 0x7fbf74005080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8affd640 1 -- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf740039a0 msgr2=0x7fbf74003d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf740039a0 0x7fbf74003d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8affd640 1 -- 172.21.15.161:0/2322762756 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf74003520 con 0x7fbf74004350 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8b7fe640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf740039a0 0x7fbf74003d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.390+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf74004820 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009ac0 tx=0x7fbf740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c788624677bd2e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8002c6b0 con 0x7fbf74004350 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf8002c810 con 0x7fbf74004350 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf8a7fc640 1 -- 172.21.15.161:0/2322762756 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf8002cba0 con 0x7fbf74004350 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf91f81640 1 -- 172.21.15.161:0/2322762756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 msgr2=0x7fbf74004820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf91f81640 1 --2- 172.21.15.161:0/2322762756 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf74004820 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009ac0 tx=0x7fbf740057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf8bfff640 1 -- 172.21.15.161:0/2322762756 reap_dead start 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf91f81640 1 -- 172.21.15.161:0/2322762756 shutdown_connections 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf91f81640 1 -- 172.21.15.161:0/2322762756 >> 172.21.15.161:0/2322762756 conn(0x7fbf8c06b080 msgr2=0x7fbf8c06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf91f81640 1 -- 172.21.15.161:0/2322762756 shutdown_connections 2024-03-23T14:37:37.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.391+0000 7fbf91f81640 1 -- 172.21.15.161:0/2322762756 wait complete. 2024-03-23T14:37:37.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.392+0000 7fbf91f81640 1 Processor -- start 2024-03-23T14:37:37.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.392+0000 7fbf91f81640 1 -- start start 2024-03-23T14:37:37.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf91f81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf8c14bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf91f81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf8c15e450 0x7fbf8c137580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf91f81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c149070 0x7fbf8c13a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf91f81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c15ea10 con 0x7fbf74004350 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf91f81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbf8c13a260 con 0x7fbf8c15e450 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf91f81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fbf8c14c050 con 0x7fbf8c149070 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf8b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf8c14bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf8b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf8c14bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60702/0 (socket says 172.21.15.161:60702) 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf8b7fe640 1 -- 172.21.15.161:0/2258140986 learned_addr learned my addr 172.21.15.161:0/2258140986 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf8c15e450 0x7fbf8c137580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf8b7fe640 1 -- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c149070 msgr2=0x7fbf8c13a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.393+0000 7fbf8bfff640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c149070 0x7fbf8c13a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf8b7fe640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c149070 0x7fbf8c13a6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf8b7fe640 1 -- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf8c15e450 msgr2=0x7fbf8c137580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf8b7fe640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf8c15e450 0x7fbf8c137580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf8b7fe640 1 -- 172.21.15.161:0/2258140986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf8c137ac0 con 0x7fbf74004350 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf8c15e450 0x7fbf8c137580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf8bfff640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c149070 0x7fbf8c13a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf8b7fe640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf8c14bb10 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e8f0 tx=0x7fbf7c00edb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf73fff640 1 -- 172.21.15.161:0/2258140986 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c005240 con 0x7fbf74004350 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf73fff640 1 -- 172.21.15.161:0/2258140986 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbf7c0053a0 con 0x7fbf74004350 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf8c139690 con 0x7fbf74004350 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf73fff640 1 -- 172.21.15.161:0/2258140986 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf7c0047e0 con 0x7fbf74004350 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.394+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf8c139c10 con 0x7fbf74004350 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.395+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf8c0735e0 con 0x7fbf74004350 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.395+0000 7fbf73fff640 1 -- 172.21.15.161:0/2258140986 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbf7c004940 con 0x7fbf74004350 2024-03-23T14:37:37.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.396+0000 7fbf73fff640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbf7806b840 0x7fbf7806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:37.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.396+0000 7fbf73fff640 1 -- 172.21.15.161:0/2258140986 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5357+0+0 (secure 0 0 0) 0x7fbf7c0899d0 con 0x7fbf74004350 2024-03-23T14:37:37.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.396+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbf7806b840 0x7fbf7806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:37.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.396+0000 7fbf8affd640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbf7806b840 0x7fbf7806dd00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009ac0 tx=0x7fbf80030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:37.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.400+0000 7fbf73fff640 1 -- 172.21.15.161:0/2258140986 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbf7c054690 con 0x7fbf74004350 2024-03-23T14:37:37.576 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:37.579+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fbf8c04f550 con 0x7fbf74004350 2024-03-23T14:37:37.713 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:37:37.716 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:37:38.264 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.268+0000 7fbf73fff640 1 -- 172.21.15.161:0/2258140986 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v51) ==== 169+0+0 (secure 0 0 0) 0x7fbf7c019070 con 0x7fbf74004350 2024-03-23T14:37:38.264 INFO:teuthology.orchestra.run.smithi161.stderr:pool 'cephfs_data' removed 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.271+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbf7806b840 msgr2=0x7fbf7806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.271+0000 7fbf91f81640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbf7806b840 0x7fbf7806dd00 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009ac0 tx=0x7fbf80030000 comp rx=0 tx=0).stop 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.271+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 msgr2=0x7fbf8c14bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf8c14bb10 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fbf7c00e8f0 tx=0x7fbf7c00edb0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 shutdown_connections 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbf7806b840 0x7fbf7806dd00 secure :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fbf80009ac0 tx=0x7fbf80030000 comp rx=0 tx=0).stop 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbf8c149070 0x7fbf8c13a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbf8c15e450 0x7fbf8c137580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 --2- 172.21.15.161:0/2258140986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbf74004350 0x7fbf8c14bb10 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 >> 172.21.15.161:0/2258140986 conn(0x7fbf8c06b080 msgr2=0x7fbf8c141cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:38.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 shutdown_connections 2024-03-23T14:37:38.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.272+0000 7fbf91f81640 1 -- 172.21.15.161:0/2258140986 wait complete. 2024-03-23T14:37:38.282 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2024-03-23T14:37:38.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.389+0000 7f207441c640 1 Processor -- start 2024-03-23T14:37:38.385 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.389+0000 7f207441c640 1 -- start start 2024-03-23T14:37:38.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f207441c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f206c1549c0 0x7f206c06e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f207441c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c06d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f207441c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f206c0b2c20 con 0x7f206c070c20 2024-03-23T14:37:38.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f207441c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f206c06ee90 con 0x7f206c1549c0 2024-03-23T14:37:38.386 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f207441c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f206c0b2a10 con 0x7f206c15b780 2024-03-23T14:37:38.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f20721b8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f206c1549c0 0x7f206c06e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f20721b8640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f206c1549c0 0x7f206c06e950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60718/0 (socket says 172.21.15.161:60718) 2024-03-23T14:37:38.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f20721b8640 1 -- 172.21.15.161:0/1892299114 learned_addr learned my addr 172.21.15.161:0/1892299114 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:38.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f20729b9640 1 --2- 172.21.15.161:0/1892299114 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c06d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f20721b8640 1 -- 172.21.15.161:0/1892299114 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 msgr2=0x7f206c06d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:38.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f20721b8640 1 --2- 172.21.15.161:0/1892299114 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c06d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f20721b8640 1 -- 172.21.15.161:0/1892299114 >> v1:172.21.15.183:6789/0 conn(0x7f206c070c20 legacy=0x7f206c15b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.387 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.390+0000 7f20721b8640 1 -- 172.21.15.161:0/1892299114 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206c0b2f00 con 0x7f206c1549c0 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.391+0000 7f20721b8640 1 --2- 172.21.15.161:0/1892299114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c06e950 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f206800d150 tx=0x7f206802e160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6560532498656648 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.391+0000 7f20711b6640 1 -- 172.21.15.161:0/1892299114 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f206800b0d0 con 0x7f206c1549c0 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.391+0000 7f20711b6640 1 -- 172.21.15.161:0/1892299114 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f206800b230 con 0x7f206c1549c0 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.391+0000 7f20711b6640 1 -- 172.21.15.161:0/1892299114 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2068009600 con 0x7f206c1549c0 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.392+0000 7f207441c640 1 -- 172.21.15.161:0/1892299114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 msgr2=0x7f206c06e950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.392+0000 7f207441c640 1 --2- 172.21.15.161:0/1892299114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c06e950 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f206800d150 tx=0x7f206802e160 comp rx=0 tx=0).stop 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.392+0000 7f207441c640 1 -- 172.21.15.161:0/1892299114 shutdown_connections 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.392+0000 7f207441c640 1 --2- 172.21.15.161:0/1892299114 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c06d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.392+0000 7f207441c640 1 --2- 172.21.15.161:0/1892299114 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c06e950 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.392+0000 7f207441c640 1 -- 172.21.15.161:0/1892299114 >> 172.21.15.161:0/1892299114 conn(0x7f206c06b4a0 msgr2=0x7f206c06b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:38.389 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.392+0000 7f207441c640 1 -- 172.21.15.161:0/1892299114 shutdown_connections 2024-03-23T14:37:38.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.392+0000 7f207441c640 1 -- 172.21.15.161:0/1892299114 wait complete. 2024-03-23T14:37:38.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.393+0000 7f207441c640 1 Processor -- start 2024-03-23T14:37:38.390 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.393+0000 7f207441c640 1 -- start start 2024-03-23T14:37:38.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f207441c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c070c20 0x7f206c1484e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f207441c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c148db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.392 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f207441c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c14b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f207441c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f206c14aba0 con 0x7f206c1549c0 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f207441c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f206c148a20 con 0x7f206c070c20 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f207441c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f206c149b80 con 0x7f206c15b780 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c148db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c148db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60728/0 (socket says 172.21.15.161:60728) 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 -- 172.21.15.161:0/2138041368 learned_addr learned my addr 172.21.15.161:0/2138041368 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20721b8640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c070c20 0x7f206c1484e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20729b9640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c14b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 -- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 msgr2=0x7f206c14b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c14b3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.393 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 -- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c070c20 msgr2=0x7f206c1484e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c070c20 0x7f206c1484e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 -- 172.21.15.161:0/2138041368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206800bd40 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20729b9640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c14b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f20719b7640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c148db0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f205c00c400 tx=0x7f205c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f2062ffd640 1 -- 172.21.15.161:0/2138041368 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c0052d0 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.394+0000 7f2062ffd640 1 -- 172.21.15.161:0/2138041368 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f205c005450 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.395+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206c149e60 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.395+0000 7f2062ffd640 1 -- 172.21.15.161:0/2138041368 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205c007260 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.395+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f206c0cca60 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.395+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f206c073570 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.396+0000 7f2062ffd640 1 -- 172.21.15.161:0/2138041368 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f205c0073c0 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.396+0000 7f2062ffd640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f205006b810 0x7f205006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.396+0000 7f2062ffd640 1 -- 172.21.15.161:0/2138041368 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f205c0891b0 con 0x7f206c1549c0 2024-03-23T14:37:38.394 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.396+0000 7f20721b8640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f205006b810 0x7f205006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.395 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.397+0000 7f20721b8640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f205006b810 0x7f205006dcd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2068037000 tx=0x7f2068032000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:38.396 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.401+0000 7f2062ffd640 1 -- 172.21.15.161:0/2138041368 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f205c054010 con 0x7f206c1549c0 2024-03-23T14:37:38.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.573+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) -- 0x7f206c04f550 con 0x7f206c1549c0 2024-03-23T14:37:38.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.576+0000 7f2062ffd640 1 -- 172.21.15.161:0/2138041368 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) ==== 104+0+0 (secure 0 0 0) 0x7f205c0576d0 con 0x7f206c1549c0 2024-03-23T14:37:38.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f205006b810 msgr2=0x7f205006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f205006b810 0x7f205006dcd0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2068037000 tx=0x7f2068032000 comp rx=0 tx=0).stop 2024-03-23T14:37:38.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 msgr2=0x7f206c148db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c148db0 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f205c00c400 tx=0x7f205c00c8c0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 shutdown_connections 2024-03-23T14:37:38.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f205006b810 0x7f205006dcd0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f206c15b780 0x7f206c14b3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f206c1549c0 0x7f206c148db0 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 --2- 172.21.15.161:0/2138041368 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f206c070c20 0x7f206c1484e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 >> 172.21.15.161:0/2138041368 conn(0x7f206c06b4a0 msgr2=0x7f206c0b0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:38.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 shutdown_connections 2024-03-23T14:37:38.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.578+0000 7f207441c640 1 -- 172.21.15.161:0/2138041368 wait complete. 2024-03-23T14:37:38.585 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm' 2024-03-23T14:37:38.685 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.690+0000 7fdd66756640 1 Processor -- start 2024-03-23T14:37:38.686 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.690+0000 7fdd66756640 1 -- start start 2024-03-23T14:37:38.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.690+0000 7fdd66756640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd601552f0 0x7fdd600b3d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.690+0000 7fdd66756640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdd600d0380 0x7fdd6006ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.690+0000 7fdd66756640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd6015f480 con 0x7fdd600b2aa0 2024-03-23T14:37:38.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.690+0000 7fdd66756640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fdd601556d0 con 0x7fdd600d0380 2024-03-23T14:37:38.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.690+0000 7fdd66756640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fdd6015e870 con 0x7fdd601552f0 2024-03-23T14:37:38.687 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd64cf3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdd600d0380 0x7fdd6006ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd5ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd601552f0 0x7fdd600b3d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd64cf3640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fdd600d0380 0x7fdd6006ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3300/0 says I am v2:172.21.15.161:41766/0 (socket says 172.21.15.161:41766) 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd5ffff640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd601552f0 0x7fdd600b3d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.183:3301/0 says I am v2:172.21.15.161:36076/0 (socket says 172.21.15.161:36076) 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd64cf3640 1 -- 172.21.15.161:0/249876795 learned_addr learned my addr 172.21.15.161:0/249876795 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd5f7fe640 1 --1- 172.21.15.161:0/249876795 >> v1:172.21.15.183:6789/0 conn(0x7fdd600b2aa0 0x7fdd600cfc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.161:52286/0 (socket says 172.21.15.161:52286) 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd64cf3640 1 -- 172.21.15.161:0/249876795 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd601552f0 msgr2=0x7fdd600b3d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd64cf3640 1 --2- 172.21.15.161:0/249876795 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd601552f0 0x7fdd600b3d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd64cf3640 1 -- 172.21.15.161:0/249876795 >> v1:172.21.15.183:6789/0 conn(0x7fdd600b2aa0 legacy=0x7fdd600cfc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd5effd640 1 -- 172.21.15.161:0/249876795 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991753350 0 0) 0x7fdd6015f480 con 0x7fdd600b2aa0 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd64cf3640 1 -- 172.21.15.161:0/249876795 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6015eb50 con 0x7fdd600d0380 2024-03-23T14:37:38.688 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.691+0000 7fdd5ffff640 1 --2- 172.21.15.161:0/249876795 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd601552f0 0x7fdd600b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.692+0000 7fdd64cf3640 1 --2- 172.21.15.161:0/249876795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 0x7fdd6006ded0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009e50 tx=0x7fdd5802bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c7fa3528d78a74d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.692+0000 7fdd5effd640 1 -- 172.21.15.161:0/249876795 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5802cb00 con 0x7fdd600d0380 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.692+0000 7fdd5effd640 1 -- 172.21.15.161:0/249876795 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd5802cc60 con 0x7fdd600d0380 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.692+0000 7fdd5effd640 1 -- 172.21.15.161:0/249876795 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd580343f0 con 0x7fdd600d0380 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.693+0000 7fdd66756640 1 -- 172.21.15.161:0/249876795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 msgr2=0x7fdd6006ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.693+0000 7fdd66756640 1 --2- 172.21.15.161:0/249876795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 0x7fdd6006ded0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdd58009e50 tx=0x7fdd5802bf60 comp rx=0 tx=0).stop 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.693+0000 7fdd66756640 1 -- 172.21.15.161:0/249876795 shutdown_connections 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.693+0000 7fdd66756640 1 --2- 172.21.15.161:0/249876795 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 0x7fdd6006ded0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.693+0000 7fdd66756640 1 --2- 172.21.15.161:0/249876795 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd601552f0 0x7fdd600b3d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.693+0000 7fdd66756640 1 -- 172.21.15.161:0/249876795 >> 172.21.15.161:0/249876795 conn(0x7fdd6006b260 msgr2=0x7fdd6006b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.693+0000 7fdd66756640 1 -- 172.21.15.161:0/249876795 shutdown_connections 2024-03-23T14:37:38.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.693+0000 7fdd66756640 1 -- 172.21.15.161:0/249876795 wait complete. 2024-03-23T14:37:38.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.694+0000 7fdd66756640 1 Processor -- start 2024-03-23T14:37:38.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.694+0000 7fdd66756640 1 -- start start 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd66756640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd600b2aa0 0x7fdd600ccdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd66756640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 0x7fdd60156b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd66756640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd601552f0 0x7fdd600af610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd66756640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fdd600cd300 con 0x7fdd601552f0 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd66756640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fdd600cd5d0 con 0x7fdd600d0380 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd66756640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fdd600cd8b0 con 0x7fdd600b2aa0 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd64cf3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd601552f0 0x7fdd600af610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd64cf3640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd601552f0 0x7fdd600af610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60730/0 (socket says 172.21.15.161:60730) 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd64cf3640 1 -- 172.21.15.161:0/895523568 learned_addr learned my addr 172.21.15.161:0/895523568 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd5f7fe640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 0x7fdd60156b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd5ffff640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd600b2aa0 0x7fdd600ccdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd64cf3640 1 -- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd600b2aa0 msgr2=0x7fdd600ccdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd64cf3640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd600b2aa0 0x7fdd600ccdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd64cf3640 1 -- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 msgr2=0x7fdd60156b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd64cf3640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 0x7fdd60156b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.695+0000 7fdd64cf3640 1 -- 172.21.15.161:0/895523568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd50009400 con 0x7fdd601552f0 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.696+0000 7fdd64cf3640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd601552f0 0x7fdd600af610 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdd58036040 tx=0x7fdd58004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.696+0000 7fdd5cff9640 1 -- 172.21.15.161:0/895523568 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58009510 con 0x7fdd601552f0 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.696+0000 7fdd5cff9640 1 -- 172.21.15.161:0/895523568 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd5803e450 con 0x7fdd601552f0 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.696+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd58009b30 con 0x7fdd601552f0 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.696+0000 7fdd5cff9640 1 -- 172.21.15.161:0/895523568 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd5803e5e0 con 0x7fdd601552f0 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.696+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd600cde70 con 0x7fdd601552f0 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.697+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd600ce390 con 0x7fdd601552f0 2024-03-23T14:37:38.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.697+0000 7fdd5cff9640 1 -- 172.21.15.161:0/895523568 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fdd5803e740 con 0x7fdd601552f0 2024-03-23T14:37:38.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.698+0000 7fdd5cff9640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd4c06b810 0x7fdd4c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:38.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.698+0000 7fdd5cff9640 1 -- 172.21.15.161:0/895523568 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7fdd580ac7d0 con 0x7fdd601552f0 2024-03-23T14:37:38.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.698+0000 7fdd5ffff640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd4c06b810 0x7fdd4c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:38.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.698+0000 7fdd5ffff640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd4c06b810 0x7fdd4c06dcd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009c60 tx=0x7fdd50008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:38.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.702+0000 7fdd5cff9640 1 -- 172.21.15.161:0/895523568 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fdd600ce390 con 0x7fdd601552f0 2024-03-23T14:37:38.940 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:38.944+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]} v 0) -- 0x7fdd600ce390 con 0x7fdd601552f0 2024-03-23T14:37:39.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.272+0000 7fdd5cff9640 1 -- 172.21.15.161:0/895523568 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_pool_perm.TestPoolPerm.test_pool_perm"]}]=0 v154) ==== 134+0+0 (secure 0 0 0) 0x7fdd600ce390 con 0x7fdd601552f0 2024-03-23T14:37:39.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.274+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd4c06b810 msgr2=0x7fdd4c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:39.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.274+0000 7fdd66756640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd4c06b810 0x7fdd4c06dcd0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fdd50009c60 tx=0x7fdd50008040 comp rx=0 tx=0).stop 2024-03-23T14:37:39.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.274+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd601552f0 msgr2=0x7fdd600af610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:39.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.274+0000 7fdd66756640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd601552f0 0x7fdd600af610 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fdd58036040 tx=0x7fdd58004a40 comp rx=0 tx=0).stop 2024-03-23T14:37:39.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.275+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 shutdown_connections 2024-03-23T14:37:39.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.275+0000 7fdd66756640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fdd4c06b810 0x7fdd4c06dcd0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:39.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.275+0000 7fdd66756640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fdd601552f0 0x7fdd600af610 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:39.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.275+0000 7fdd66756640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fdd600d0380 0x7fdd60156b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:39.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.275+0000 7fdd66756640 1 --2- 172.21.15.161:0/895523568 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fdd600b2aa0 0x7fdd600ccdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:39.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.275+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 >> 172.21.15.161:0/895523568 conn(0x7fdd6006b260 msgr2=0x7fdd6015b210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:39.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.275+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 shutdown_connections 2024-03-23T14:37:39.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:39.275+0000 7fdd66756640 1 -- 172.21.15.161:0/895523568 wait complete. 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner:test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) ... ERROR 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner: 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-23T14:37:39.283 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi161 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner: 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner:Ran 2 tests in 121.478s 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner: 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner:FAILED (errors=1) 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner: 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner:====================================================================== 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner:ERROR: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-23T14:37:39.284 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/test_pool_perm.py", line 52, in test_pool_perm 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: self.mount_a.run_python(remote_script.format(path=file_path, check_read=str(True))) 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs/mount.py", line 776, in run_python 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: p.wait() 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: self._raise_for_status() 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: raise CommandFailedError( 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner:teuthology.exceptions.CommandFailedError: Command failed on smithi161 with status 1: 'stdin-killer -- python3 -c \'\nimport os\nimport errno\n\nfd = os.open("/home/ubuntu/cephtest/mnt.0/test_file", os.O_RDWR)\ntry:\n if True:\n ret = os.read(fd, 1024)\n else:\n os.write(fd, b\'"\'"\'content\'"\'"\')\nexcept OSError as e:\n if e.errno != errno.EPERM:\n raise\nelse:\n raise RuntimeError("client does not check permission of data pool")\n\'' 2024-03-23T14:37:39.285 INFO:tasks.cephfs_test_runner: 2024-03-23T14:37:39.285 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-23T14:37:39.622 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=f278c6e9690a4392ae3c720a3818398d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_batrick_ceph_d39ab7bb1f1162ec7b83b4f13a6bb876d4baf21b/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm) 2024-03-23T14:37:39.624 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2024-03-23T14:37:39.635 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-03-23T14:37:39.678 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-03-23T14:37:39.679 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:37:39.679 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:39.698 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18767 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T14:37:39.698 DEBUG:teuthology.orchestra.run.smithi161:> set -ex 2024-03-23T14:37:39.698 DEBUG:teuthology.orchestra.run.smithi161:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:39.754 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056372,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=18767 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248844k,nr_inodes=812211,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-0 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-1 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-2 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-3 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497692k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.2 nsfs rw 0 0 2024-03-23T14:37:39.754 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:37:39.754 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:39.773 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056359,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17751 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T14:37:39.773 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-23T14:37:39.773 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-23T14:37:39.832 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056359,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17751 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248832k,nr_inodes=812208,mode=700,uid=1000,gid=1301,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-4 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-5 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-6 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-7 xfs rw,seclabel,noatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497672k,nr_inodes=819200,mode=755,inode64 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.3 nsfs rw 0 0 2024-03-23T14:37:39.832 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:39.832 DEBUG:teuthology.orchestra.run.smithi161:> ip netns list 2024-03-23T14:37:39.847 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.2 (id: 1) 2024-03-23T14:37:39.847 INFO:teuthology.orchestra.run.smithi161.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-23T14:37:39.848 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:39.848 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.2 2024-03-23T14:37:39.914 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:39.915 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-23T14:37:39.982 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:39.983 DEBUG:teuthology.orchestra.run.smithi161:> sudo ip link delete ceph-brx 2024-03-23T14:37:40.073 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:40.073 DEBUG:teuthology.orchestra.run.smithi183:> ip netns list 2024-03-23T14:37:40.092 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-ns--home-ubuntu-cephtest-mnt.3 (id: 1) 2024-03-23T14:37:40.092 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-ns--home-ubuntu-cephtest-mnt.1 (id: 0) 2024-03-23T14:37:40.092 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:40.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.3 2024-03-23T14:37:40.164 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:40.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-23T14:37:40.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-23T14:37:40.235 DEBUG:teuthology.orchestra.run.smithi183:> sudo ip link delete ceph-brx 2024-03-23T14:37:40.331 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-23T14:37:40.341 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-03-23T14:37:40.341 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:37:40.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.449+0000 7fbc32047640 1 Processor -- start 2024-03-23T14:37:40.445 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.449+0000 7fbc32047640 1 -- start start 2024-03-23T14:37:40.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc32047640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc2c145080 0x7fbc2c072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc32047640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c1607d0 0x7fbc2c15fae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.446 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc32047640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fbc2c06cbf0 0x7fbc2c160020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc32047640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fbc2c071010 con 0x7fbc2c145080 2024-03-23T14:37:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc32047640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fbc2c0cf4f0 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc32047640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fbc2c0cef30 con 0x7fbc2c1607d0 2024-03-23T14:37:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc2b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc2c145080 0x7fbc2c072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc2b7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fbc2c145080 0x7fbc2c072a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60732/0 (socket says 172.21.15.161:60732) 2024-03-23T14:37:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc2b7fe640 1 -- 172.21.15.161:0/43705169 learned_addr learned my addr 172.21.15.161:0/43705169 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc2b7fe640 1 -- 172.21.15.161:0/43705169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c1607d0 msgr2=0x7fbc2c15fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:40.447 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc2b7fe640 1 --2- 172.21.15.161:0/43705169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c1607d0 0x7fbc2c15fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc2b7fe640 1 -- 172.21.15.161:0/43705169 >> v2:172.21.15.183:3300/0 conn(0x7fbc2c06cbf0 msgr2=0x7fbc2c160020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:40.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc2b7fe640 1 --2- 172.21.15.161:0/43705169 >> v2:172.21.15.183:3300/0 conn(0x7fbc2c06cbf0 0x7fbc2c160020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.448 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.450+0000 7fbc2b7fe640 1 -- 172.21.15.161:0/43705169 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc2c0cf210 con 0x7fbc2c145080 2024-03-23T14:37:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.451+0000 7fbc2b7fe640 1 --2- 172.21.15.161:0/43705169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c145080 0x7fbc2c072a00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0734c0 tx=0x7fbc1c02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb07f1a4c0f667cc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.451+0000 7fbc2a7fc640 1 -- 172.21.15.161:0/43705169 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c0317e0 con 0x7fbc2c145080 2024-03-23T14:37:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.451+0000 7fbc2a7fc640 1 -- 172.21.15.161:0/43705169 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1c031940 con 0x7fbc2c145080 2024-03-23T14:37:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.451+0000 7fbc2a7fc640 1 -- 172.21.15.161:0/43705169 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c031cd0 con 0x7fbc2c145080 2024-03-23T14:37:40.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 -- 172.21.15.161:0/43705169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c145080 msgr2=0x7fbc2c072a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 --2- 172.21.15.161:0/43705169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c145080 0x7fbc2c072a00 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c0734c0 tx=0x7fbc1c02b880 comp rx=0 tx=0).stop 2024-03-23T14:37:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 -- 172.21.15.161:0/43705169 shutdown_connections 2024-03-23T14:37:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 --2- 172.21.15.161:0/43705169 >> v2:172.21.15.183:3300/0 conn(0x7fbc2c06cbf0 0x7fbc2c160020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 --2- 172.21.15.161:0/43705169 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c1607d0 0x7fbc2c15fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 --2- 172.21.15.161:0/43705169 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c145080 0x7fbc2c072a00 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 -- 172.21.15.161:0/43705169 >> 172.21.15.161:0/43705169 conn(0x7fbc2c0b17f0 msgr2=0x7fbc2c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 -- 172.21.15.161:0/43705169 shutdown_connections 2024-03-23T14:37:40.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.452+0000 7fbc32047640 1 -- 172.21.15.161:0/43705169 wait complete. 2024-03-23T14:37:40.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.453+0000 7fbc32047640 1 Processor -- start 2024-03-23T14:37:40.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.453+0000 7fbc32047640 1 -- start start 2024-03-23T14:37:40.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc32047640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c06cbf0 0x7fbc2c0a9980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc32047640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c145080 0x7fbc2c13cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc32047640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c1607d0 0x7fbc2c0ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc32047640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c13d530 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc32047640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fbc2c13d800 con 0x7fbc2c1607d0 2024-03-23T14:37:40.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc32047640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fbc2c13dae0 con 0x7fbc2c145080 2024-03-23T14:37:40.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c06cbf0 0x7fbc2c0a9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c06cbf0 0x7fbc2c0a9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:60734/0 (socket says 172.21.15.161:60734) 2024-03-23T14:37:40.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 -- 172.21.15.161:0/1032862742 learned_addr learned my addr 172.21.15.161:0/1032862742 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:40.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2bfff640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c1607d0 0x7fbc2c0ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2affd640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c145080 0x7fbc2c13cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 -- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c145080 msgr2=0x7fbc2c13cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c145080 0x7fbc2c13cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 -- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c1607d0 msgr2=0x7fbc2c0ae7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c1607d0 0x7fbc2c0ae7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 -- 172.21.15.161:0/1032862742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc1c0094d0 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2affd640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c145080 0x7fbc2c13cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:40.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.454+0000 7fbc2b7fe640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c06cbf0 0x7fbc2c0a9980 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c02b8c0 tx=0x7fbc1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:40.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.455+0000 7fbc13fff640 1 -- 172.21.15.161:0/1032862742 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c032940 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.455+0000 7fbc13fff640 1 -- 172.21.15.161:0/1032862742 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbc1c032aa0 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.455+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc2c13ddc0 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.455+0000 7fbc13fff640 1 -- 172.21.15.161:0/1032862742 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc1c032e50 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.455+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc2c13e380 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.456+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc2c0735e0 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.456+0000 7fbc13fff640 1 -- 172.21.15.161:0/1032862742 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fbc1c04e450 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.457+0000 7fbc13fff640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbc1806b810 0x7fbc1806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.457+0000 7fbc13fff640 1 -- 172.21.15.161:0/1032862742 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7fbc1c0abff0 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.457+0000 7fbc2affd640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbc1806b810 0x7fbc1806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.457+0000 7fbc2affd640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbc1806b810 0x7fbc1806dcd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c142400 tx=0x7fbc20008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:40.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.461+0000 7fbc13fff640 1 -- 172.21.15.161:0/1032862742 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fbc1c076ec0 con 0x7fbc2c06cbf0 2024-03-23T14:37:40.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.630+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbc2c0cec00 con 0x7fbc1806b810 2024-03-23T14:37:40.626 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.631+0000 7fbc13fff640 1 -- 172.21.15.161:0/1032862742 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13490 (secure 0 0 0) 0x7fbc20009480 con 0x7fbc1806b810 2024-03-23T14:37:40.627 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:40.627 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:37:40.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbc1806b810 msgr2=0x7fbc1806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbc1806b810 0x7fbc1806dcd0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbc2c142400 tx=0x7fbc20008040 comp rx=0 tx=0).stop 2024-03-23T14:37:40.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c06cbf0 msgr2=0x7fbc2c0a9980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c06cbf0 0x7fbc2c0a9980 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c02b8c0 tx=0x7fbc1c0046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 shutdown_connections 2024-03-23T14:37:40.629 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fbc1806b810 0x7fbc1806dcd0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fbc2c1607d0 0x7fbc2c0ae7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fbc2c145080 0x7fbc2c13cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 --2- 172.21.15.161:0/1032862742 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fbc2c06cbf0 0x7fbc2c0a9980 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 >> 172.21.15.161:0/1032862742 conn(0x7fbc2c0b17f0 msgr2=0x7fbc2c0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:40.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 shutdown_connections 2024-03-23T14:37:40.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.633+0000 7fbc32047640 1 -- 172.21.15.161:0/1032862742 wait complete. 2024-03-23T14:37:40.639 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-03-23T14:37:40.505435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":222072,"kb_used_data":5900,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752648,"statfs":{"total":773094113280,"available":772866711552,"internally_reserved":0,"allocated":6041600,"data_stored":3193428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14792,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-66,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001328"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-23T14:37:40.006904+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:37:40.006904+0000","last_peered":"2024-03-23T14:37:40.006904+0000","last_clean":"2024-03-23T14:37:40.006904+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:37:40.006904+0000","last_undegraded":"2024-03-23T14:37:40.006904+0000","last_fullsized":"2024-03-23T14:37:40.006904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":839154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":585728,"data_stored":243365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":248786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":844719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:37:40.640 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:37:40.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.746+0000 7f49da464640 1 Processor -- start 2024-03-23T14:37:40.742 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.746+0000 7f49da464640 1 -- start start 2024-03-23T14:37:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.746+0000 7f49da464640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f49d40d1470 0x7f49d40d2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.746+0000 7f49da464640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d40d2970 0x7f49d40b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49da464640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d40d0460 con 0x7f49d415e400 2024-03-23T14:37:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49da464640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f49d40d1050 con 0x7f49d40d1470 2024-03-23T14:37:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49da464640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f49d40d2e00 con 0x7f49d40d2970 2024-03-23T14:37:40.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d3fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f49d415e400 0x7f49d40cf490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41006/0 (socket says 172.21.15.161:41006) 2024-03-23T14:37:40.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d3fff640 1 -- 172.21.15.161:0/3032261463 learned_addr learned my addr 172.21.15.161:0/3032261463 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:40.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49cbfff640 1 --2- 172.21.15.161:0/3032261463 >> v2:172.21.15.183:3300/0 conn(0x7f49d40d1470 0x7f49d40d2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d8a01640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d40d2970 0x7f49d40b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2518824338 0 0) 0x7f49d40d0460 con 0x7f49d415e400 2024-03-23T14:37:40.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f49c0003520 con 0x7f49d415e400 2024-03-23T14:37:40.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1697164173 0 0) 0x7f49c0003520 con 0x7f49d415e400 2024-03-23T14:37:40.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d40d2970 msgr2=0x7f49d40b0a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:40.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d40d2970 0x7f49d40b0a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 >> v2:172.21.15.183:3300/0 conn(0x7f49d40d1470 msgr2=0x7f49d40d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:40.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 --2- 172.21.15.161:0/3032261463 >> v2:172.21.15.183:3300/0 conn(0x7f49d40d1470 0x7f49d40d2430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d40d06b0 con 0x7f49d415e400 2024-03-23T14:37:40.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f49c4001670 con 0x7f49d415e400 2024-03-23T14:37:40.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49cbfff640 1 --2- 172.21.15.161:0/3032261463 >> v2:172.21.15.183:3300/0 conn(0x7f49d40d1470 0x7f49d40d2430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:40.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 >> v1:172.21.15.161:6789/0 conn(0x7f49d415e400 legacy=0x7f49d40cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49c0003970 0x7f49c0003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d40d2970 0x7f49c00047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c0004d00 0x7f49c0005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49d40d1050 con 0x7f49c0004d00 2024-03-23T14:37:40.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f49d40d2e00 con 0x7f49c0003970 2024-03-23T14:37:40.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.747+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f49c000b980 con 0x7f49d40d2970 2024-03-23T14:37:40.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.748+0000 7f49d8a01640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c0004d00 0x7f49c0005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.748+0000 7f49d3fff640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49c0003970 0x7f49c0003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.748+0000 7f49cbfff640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d40d2970 0x7f49c00047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.748+0000 7f49d8a01640 1 -- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d40d2970 msgr2=0x7f49c00047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:40.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.748+0000 7f49d8a01640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d40d2970 0x7f49c00047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.748+0000 7f49d8a01640 1 -- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49c0003970 msgr2=0x7f49c0003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:40.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.748+0000 7f49d8a01640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49c0003970 0x7f49c0003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.748+0000 7f49d8a01640 1 -- 172.21.15.161:0/3032261463 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49c0003520 con 0x7f49c0004d00 2024-03-23T14:37:40.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49d8a01640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c0004d00 0x7f49c0005180 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00e170 tx=0x7f49c00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=907611e0ad496c1e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:40.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc02cbf0 con 0x7f49c0004d00 2024-03-23T14:37:40.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49cc00f040 con 0x7f49c0004d00 2024-03-23T14:37:40.750 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49d37fe640 1 -- 172.21.15.161:0/3032261463 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49cc034440 con 0x7f49c0004d00 2024-03-23T14:37:40.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49da464640 1 -- 172.21.15.161:0/3032261463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c0004d00 msgr2=0x7f49c0005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49da464640 1 --2- 172.21.15.161:0/3032261463 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49c0004d00 0x7f49c0005180 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f49cc00e170 tx=0x7f49c00057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49d8a01640 1 -- 172.21.15.161:0/3032261463 reap_dead start 2024-03-23T14:37:40.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49da464640 1 -- 172.21.15.161:0/3032261463 shutdown_connections 2024-03-23T14:37:40.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49da464640 1 -- 172.21.15.161:0/3032261463 >> 172.21.15.161:0/3032261463 conn(0x7f49d409d190 msgr2=0x7f49d40d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:40.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49da464640 1 -- 172.21.15.161:0/3032261463 shutdown_connections 2024-03-23T14:37:40.751 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.749+0000 7f49da464640 1 -- 172.21.15.161:0/3032261463 wait complete. 2024-03-23T14:37:40.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49da464640 1 Processor -- start 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49da464640 1 -- start start 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49da464640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49d41f3620 0x7f49d41f3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49da464640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d4160a60 0x7f49d4160ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49da464640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49d40b3210 0x7f49d40b36a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49da464640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f49d40b2dc0 con 0x7f49d41f3620 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49da464640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f49d4161420 con 0x7f49d40b3210 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49da464640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f49d41f40f0 con 0x7f49d4160a60 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49d41f3620 0x7f49d41f3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49d3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49d41f3620 0x7f49d41f3a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38024/0 (socket says 172.21.15.161:38024) 2024-03-23T14:37:40.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49d3fff640 1 -- 172.21.15.161:0/683683775 learned_addr learned my addr 172.21.15.161:0/683683775 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:40.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.751+0000 7f49d8a01640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49d40b3210 0x7f49d40b36a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49cbfff640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d4160a60 0x7f49d4160ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d3fff640 1 -- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d4160a60 msgr2=0x7f49d4160ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d3fff640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d4160a60 0x7f49d4160ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d3fff640 1 -- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49d40b3210 msgr2=0x7f49d40b36a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d3fff640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49d40b3210 0x7f49d40b36a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d3fff640 1 -- 172.21.15.161:0/683683775 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d41f43d0 con 0x7f49d41f3620 2024-03-23T14:37:40.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d8a01640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49d40b3210 0x7f49d40b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:40.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d3fff640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49d41f3620 0x7f49d41f3a00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f49c4004840 tx=0x7f49c4007720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:40.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d17fa640 1 -- 172.21.15.161:0/683683775 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c400f960 con 0x7f49d41f3620 2024-03-23T14:37:40.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d17fa640 1 -- 172.21.15.161:0/683683775 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c400fac0 con 0x7f49d41f3620 2024-03-23T14:37:40.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d41f46b0 con 0x7f49d41f3620 2024-03-23T14:37:40.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49d17fa640 1 -- 172.21.15.161:0/683683775 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c400fe70 con 0x7f49d41f3620 2024-03-23T14:37:40.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.752+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d40b3be0 con 0x7f49d41f3620 2024-03-23T14:37:40.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.753+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d40a1e00 con 0x7f49d41f3620 2024-03-23T14:37:40.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.753+0000 7f49d17fa640 1 -- 172.21.15.161:0/683683775 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f49c401d460 con 0x7f49d41f3620 2024-03-23T14:37:40.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.754+0000 7f49d17fa640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f49b406b810 0x7f49b406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:40.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.754+0000 7f49d17fa640 1 -- 172.21.15.161:0/683683775 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f49c408a140 con 0x7f49d41f3620 2024-03-23T14:37:40.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.754+0000 7f49cbfff640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f49b406b810 0x7f49b406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:40.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.754+0000 7f49cbfff640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f49b406b810 0x7f49b406dcd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009d10 tx=0x7f49bc008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:40.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.758+0000 7f49d17fa640 1 -- 172.21.15.161:0/683683775 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f49c4058990 con 0x7f49d41f3620 2024-03-23T14:37:40.920 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.925+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f49d40d0c70 con 0x7f49b406b810 2024-03-23T14:37:40.921 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.926+0000 7f49d17fa640 1 -- 172.21.15.161:0/683683775 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13490 (secure 0 0 0) 0x7f49bc009960 con 0x7f49b406b810 2024-03-23T14:37:40.922 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:40.922 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:37:40.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f49b406b810 msgr2=0x7f49b406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f49b406b810 0x7f49b406dcd0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009d10 tx=0x7f49bc008040 comp rx=0 tx=0).stop 2024-03-23T14:37:40.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49d41f3620 msgr2=0x7f49d41f3a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:40.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49d41f3620 0x7f49d41f3a00 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f49c4004840 tx=0x7f49c4007720 comp rx=0 tx=0).stop 2024-03-23T14:37:40.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 shutdown_connections 2024-03-23T14:37:40.924 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f49b406b810 0x7f49b406dcd0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f49d40b3210 0x7f49d40b36a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f49d4160a60 0x7f49d4160ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 --2- 172.21.15.161:0/683683775 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f49d41f3620 0x7f49d41f3a00 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:40.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 >> 172.21.15.161:0/683683775 conn(0x7f49d409d190 msgr2=0x7f49d40b6210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:40.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 shutdown_connections 2024-03-23T14:37:40.925 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:40.928+0000 7f49da464640 1 -- 172.21.15.161:0/683683775 wait complete. 2024-03-23T14:37:40.935 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-03-23T14:37:40.505435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":222072,"kb_used_data":5900,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752648,"statfs":{"total":773094113280,"available":772866711552,"internally_reserved":0,"allocated":6041600,"data_stored":3193428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14792,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-66,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001328"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-23T14:37:40.006904+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:37:40.006904+0000","last_peered":"2024-03-23T14:37:40.006904+0000","last_clean":"2024-03-23T14:37:40.006904+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:37:40.006904+0000","last_undegraded":"2024-03-23T14:37:40.006904+0000","last_fullsized":"2024-03-23T14:37:40.006904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":839154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":585728,"data_stored":243365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":248786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":844719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:37:40.935 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-03-23T14:37:40.935 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:37:41.036 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.040+0000 7f89378d0640 1 Processor -- start 2024-03-23T14:37:41.037 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.041+0000 7f89378d0640 1 -- start start 2024-03-23T14:37:41.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.041+0000 7f89378d0640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f89300724e0 0x7f893006e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.041+0000 7f89378d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f89300b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.041+0000 7f89378d0640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f893015be70 con 0x7f8930072100 2024-03-23T14:37:41.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.041+0000 7f89378d0640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f893006ef80 con 0x7f89300724e0 2024-03-23T14:37:41.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.041+0000 7f89378d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f893015c430 con 0x7f893006ead0 2024-03-23T14:37:41.040 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.041+0000 7f893566c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f8930072100 0x7f893015b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41014/0 (socket says 172.21.15.161:41014) 2024-03-23T14:37:41.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.041+0000 7f893566c640 1 -- 172.21.15.161:0/1316246168 learned_addr learned my addr 172.21.15.161:0/1316246168 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:41.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8934e6b640 1 --2- 172.21.15.161:0/1316246168 >> v2:172.21.15.183:3300/0 conn(0x7f89300724e0 0x7f893006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8935e6d640 1 --2- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f89300b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8927fff640 1 -- 172.21.15.161:0/1316246168 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031451273 0 0) 0x7f893015be70 con 0x7f8930072100 2024-03-23T14:37:41.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8927fff640 1 -- 172.21.15.161:0/1316246168 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f891c003520 con 0x7f8930072100 2024-03-23T14:37:41.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8934e6b640 1 -- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 msgr2=0x7f89300b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:41.041 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8927fff640 1 -- 172.21.15.161:0/1316246168 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 936456239 0 0) 0x7f891c003520 con 0x7f8930072100 2024-03-23T14:37:41.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8934e6b640 1 --2- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f89300b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8934e6b640 1 -- 172.21.15.161:0/1316246168 >> v1:172.21.15.161:6789/0 conn(0x7f8930072100 legacy=0x7f893015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8934e6b640 1 -- 172.21.15.161:0/1316246168 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f893015c150 con 0x7f89300724e0 2024-03-23T14:37:41.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.042+0000 7f8935e6d640 1 --2- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f89300b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:41.042 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f8934e6b640 1 --2- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89300724e0 0x7f893006e590 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f89200097a0 tx=0x7f892002b790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=16108d93d93c0111 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:41.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f8927fff640 1 -- 172.21.15.161:0/1316246168 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89280019f0 con 0x7f89300724e0 2024-03-23T14:37:41.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f8927fff640 1 -- 172.21.15.161:0/1316246168 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f892002e050 con 0x7f89300724e0 2024-03-23T14:37:41.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f8927fff640 1 -- 172.21.15.161:0/1316246168 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8920030730 con 0x7f89300724e0 2024-03-23T14:37:41.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f89378d0640 1 -- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89300724e0 msgr2=0x7f893006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f89378d0640 1 --2- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89300724e0 0x7f893006e590 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f89200097a0 tx=0x7f892002b790 comp rx=0 tx=0).stop 2024-03-23T14:37:41.043 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f89378d0640 1 -- 172.21.15.161:0/1316246168 shutdown_connections 2024-03-23T14:37:41.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f89378d0640 1 --2- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f89300b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f89378d0640 1 --2- 172.21.15.161:0/1316246168 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f89300724e0 0x7f893006e590 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.043+0000 7f89378d0640 1 -- 172.21.15.161:0/1316246168 >> 172.21.15.161:0/1316246168 conn(0x7f89300b17f0 msgr2=0x7f89300b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:41.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.044+0000 7f89378d0640 1 -- 172.21.15.161:0/1316246168 shutdown_connections 2024-03-23T14:37:41.044 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.044+0000 7f89378d0640 1 -- 172.21.15.161:0/1316246168 wait complete. 2024-03-23T14:37:41.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f89378d0640 1 Processor -- start 2024-03-23T14:37:41.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f89378d0640 1 -- start start 2024-03-23T14:37:41.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f89378d0640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f893014a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.047 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f89378d0640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8930072100 0x7f8930151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f89378d0640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89300724e0 0x7f8930153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f89378d0640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f893014c2f0 con 0x7f89300724e0 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f89378d0640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f893014cbb0 con 0x7f8930072100 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f89378d0640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f893014c560 con 0x7f893006ead0 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.045+0000 7f8935e6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89300724e0 0x7f8930153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8935e6d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89300724e0 0x7f8930153b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38034/0 (socket says 172.21.15.161:38034) 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8935e6d640 1 -- 172.21.15.161:0/175215830 learned_addr learned my addr 172.21.15.161:0/175215830 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8934e6b640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8930072100 0x7f8930151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.048 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f893566c640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f893014a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8935e6d640 1 -- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 msgr2=0x7f893014a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8935e6d640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f893014a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8935e6d640 1 -- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8930072100 msgr2=0x7f8930151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8935e6d640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8930072100 0x7f8930151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8935e6d640 1 -- 172.21.15.161:0/175215830 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f892c009400 con 0x7f89300724e0 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8935e6d640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89300724e0 0x7f8930153b40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f892c00e880 tx=0x7f892c00ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8925ffb640 1 -- 172.21.15.161:0/175215830 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c009650 con 0x7f89300724e0 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8925ffb640 1 -- 172.21.15.161:0/175215830 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f892c004020 con 0x7f89300724e0 2024-03-23T14:37:41.049 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f8925ffb640 1 -- 172.21.15.161:0/175215830 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f892c0043b0 con 0x7f89300724e0 2024-03-23T14:37:41.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8920009400 con 0x7f89300724e0 2024-03-23T14:37:41.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.046+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8930152280 con 0x7f89300724e0 2024-03-23T14:37:41.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.047+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89300735e0 con 0x7f89300724e0 2024-03-23T14:37:41.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.048+0000 7f8925ffb640 1 -- 172.21.15.161:0/175215830 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f892c00c940 con 0x7f89300724e0 2024-03-23T14:37:41.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.048+0000 7f8925ffb640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f891406b810 0x7f891406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.048+0000 7f8925ffb640 1 -- 172.21.15.161:0/175215830 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f892c089c50 con 0x7f89300724e0 2024-03-23T14:37:41.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.048+0000 7f893566c640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f891406b810 0x7f891406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.050 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.049+0000 7f893566c640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f891406b810 0x7f891406dcd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f892800f440 tx=0x7f8928009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:41.051 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.052+0000 7f8925ffb640 1 -- 172.21.15.161:0/175215830 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f892c08f020 con 0x7f89300724e0 2024-03-23T14:37:41.215 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.219+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f893015bb00 con 0x7f891406b810 2024-03-23T14:37:41.216 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.221+0000 7f8925ffb640 1 -- 172.21.15.161:0/175215830 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13490 (secure 0 0 0) 0x7f89280133b0 con 0x7f891406b810 2024-03-23T14:37:41.217 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:41.217 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:37:41.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f891406b810 msgr2=0x7f891406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f891406b810 0x7f891406dcd0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f892800f440 tx=0x7f8928009040 comp rx=0 tx=0).stop 2024-03-23T14:37:41.219 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89300724e0 msgr2=0x7f8930153b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89300724e0 0x7f8930153b40 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f892c00e880 tx=0x7f892c00ed40 comp rx=0 tx=0).stop 2024-03-23T14:37:41.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 shutdown_connections 2024-03-23T14:37:41.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f891406b810 0x7f891406dcd0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f89300724e0 0x7f8930153b40 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f8930072100 0x7f8930151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 --2- 172.21.15.161:0/175215830 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f893006ead0 0x7f893014a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.220 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 >> 172.21.15.161:0/175215830 conn(0x7f89300b17f0 msgr2=0x7f89300b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:41.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 shutdown_connections 2024-03-23T14:37:41.221 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.223+0000 7f89378d0640 1 -- 172.21.15.161:0/175215830 wait complete. 2024-03-23T14:37:41.229 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":124,"stamp":"2024-03-23T14:37:40.505435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":143,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":754974720,"kb_used":222072,"kb_used_data":5900,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752648,"statfs":{"total":773094113280,"available":772866711552,"internally_reserved":0,"allocated":6041600,"data_stored":3193428,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-14792,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-66,"num_write_kb":-37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001328"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-23T14:37:40.006904+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:37:40.006904+0000","last_peered":"2024-03-23T14:37:40.006904+0000","last_clean":"2024-03-23T14:37:40.006904+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:37:40.006904+0000","last_undegraded":"2024-03-23T14:37:40.006904+0000","last_fullsized":"2024-03-23T14:37:40.006904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":839154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574884,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27592,"kb_used_data":572,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344248,"statfs":{"total":96636764160,"available":96608509952,"internally_reserved":0,"allocated":585728,"data_stored":243365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":248786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":844719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:37:41.230 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-23T14:37:41.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.335+0000 7fda64993640 1 Processor -- start 2024-03-23T14:37:41.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.335+0000 7fda64993640 1 -- start start 2024-03-23T14:37:41.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.336+0000 7fda64993640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda60072100 0x7fda6015b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.336+0000 7fda64993640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda6015be70 con 0x7fda6006eca0 2024-03-23T14:37:41.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.336+0000 7fda64993640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fda6015c150 con 0x7fda600724e0 2024-03-23T14:37:41.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.336+0000 7fda64993640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fda6015c430 con 0x7fda60072100 2024-03-23T14:37:41.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.337+0000 7fda5e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda60072100 0x7fda6015b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.337+0000 7fda5ed9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fda6006eca0 0x7fda600b3470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41020/0 (socket says 172.21.15.161:41020) 2024-03-23T14:37:41.333 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.337+0000 7fda5e59c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fda60072100 0x7fda6015b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38036/0 (socket says 172.21.15.161:38036) 2024-03-23T14:37:41.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.337+0000 7fda5ed9d640 1 -- 172.21.15.161:0/1125501086 learned_addr learned my addr 172.21.15.161:0/1125501086 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:41.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.337+0000 7fda5e59c640 1 -- 172.21.15.161:0/1125501086 >> v1:172.21.15.183:6789/0 conn(0x7fda600724e0 legacy=0x7fda6006e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.337+0000 7fda5e59c640 1 -- 172.21.15.161:0/1125501086 >> v1:172.21.15.161:6789/0 conn(0x7fda6006eca0 legacy=0x7fda600b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.337+0000 7fda5e59c640 1 -- 172.21.15.161:0/1125501086 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda600b2c70 con 0x7fda60072100 2024-03-23T14:37:41.334 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.337+0000 7fda5d59a640 1 -- 172.21.15.161:0/1125501086 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1637672703 0 0) 0x7fda6015be70 con 0x7fda6006eca0 2024-03-23T14:37:41.335 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda5e59c640 1 --2- 172.21.15.161:0/1125501086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda6015b3f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fda540097a0 tx=0x7fda5402b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d97c2347640c9bf server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:41.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda5d59a640 1 -- 172.21.15.161:0/1125501086 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54031830 con 0x7fda60072100 2024-03-23T14:37:41.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda5d59a640 1 -- 172.21.15.161:0/1125501086 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda54031990 con 0x7fda60072100 2024-03-23T14:37:41.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda5d59a640 1 -- 172.21.15.161:0/1125501086 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda54031d20 con 0x7fda60072100 2024-03-23T14:37:41.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda64993640 1 -- 172.21.15.161:0/1125501086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 msgr2=0x7fda6015b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda64993640 1 --2- 172.21.15.161:0/1125501086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda6015b3f0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7fda540097a0 tx=0x7fda5402b8d0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda64993640 1 -- 172.21.15.161:0/1125501086 shutdown_connections 2024-03-23T14:37:41.336 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda64993640 1 --2- 172.21.15.161:0/1125501086 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda6015b3f0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda64993640 1 -- 172.21.15.161:0/1125501086 >> 172.21.15.161:0/1125501086 conn(0x7fda600b17f0 msgr2=0x7fda600b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:41.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda64993640 1 -- 172.21.15.161:0/1125501086 shutdown_connections 2024-03-23T14:37:41.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.338+0000 7fda64993640 1 -- 172.21.15.161:0/1125501086 wait complete. 2024-03-23T14:37:41.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.340+0000 7fda64993640 1 Processor -- start 2024-03-23T14:37:41.337 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.340+0000 7fda64993640 1 -- start start 2024-03-23T14:37:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.340+0000 7fda64993640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda6006eca0 0x7fda6014ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.340+0000 7fda64993640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda60153e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.340+0000 7fda64993640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda600724e0 0x7fda60154380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.340+0000 7fda64993640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fda601e3590 con 0x7fda60072100 2024-03-23T14:37:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.340+0000 7fda64993640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fda600b0c00 con 0x7fda6006eca0 2024-03-23T14:37:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.340+0000 7fda64993640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fda600b0ea0 con 0x7fda600724e0 2024-03-23T14:37:41.340 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda60153e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda60153e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38042/0 (socket says 172.21.15.161:38042) 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 -- 172.21.15.161:0/3068007985 learned_addr learned my addr 172.21.15.161:0/3068007985 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5e59c640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda6006eca0 0x7fda6014ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5ed9d640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda600724e0 0x7fda60154380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 -- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda600724e0 msgr2=0x7fda60154380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda600724e0 0x7fda60154380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 -- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda6006eca0 msgr2=0x7fda6014ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda6006eca0 0x7fda6014ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.341 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 -- 172.21.15.161:0/3068007985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda54009400 con 0x7fda60072100 2024-03-23T14:37:41.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda5dd9b640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda60153e40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c3c0 tx=0x7fda5000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:41.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda4effd640 1 -- 172.21.15.161:0/3068007985 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda500052d0 con 0x7fda60072100 2024-03-23T14:37:41.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda4effd640 1 -- 172.21.15.161:0/3068007985 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fda50005450 con 0x7fda60072100 2024-03-23T14:37:41.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda601e7530 con 0x7fda60072100 2024-03-23T14:37:41.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda4effd640 1 -- 172.21.15.161:0/3068007985 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda50003830 con 0x7fda60072100 2024-03-23T14:37:41.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.341+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda601e71c0 con 0x7fda60072100 2024-03-23T14:37:41.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.342+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fda600735e0 con 0x7fda60072100 2024-03-23T14:37:41.342 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.343+0000 7fda4effd640 1 -- 172.21.15.161:0/3068007985 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fda50003990 con 0x7fda60072100 2024-03-23T14:37:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.343+0000 7fda4effd640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda4006b810 0x7fda4006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.343+0000 7fda4effd640 1 -- 172.21.15.161:0/3068007985 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7fda50089350 con 0x7fda60072100 2024-03-23T14:37:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.343+0000 7fda5e59c640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda4006b810 0x7fda4006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.344+0000 7fda5e59c640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda4006b810 0x7fda4006dcd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fda54036000 tx=0x7fda5403c040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:41.343 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.347+0000 7fda4effd640 1 -- 172.21.15.161:0/3068007985 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fda50054120 con 0x7fda60072100 2024-03-23T14:37:41.518 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.522+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fda600cbe90 con 0x7fda60072100 2024-03-23T14:37:41.519 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.523+0000 7fda4effd640 1 -- 172.21.15.161:0/3068007985 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) ==== 74+0+13058 (secure 0 0 0) 0x7fda500577e0 con 0x7fda60072100 2024-03-23T14:37:41.519 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:41.519 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":51,"fsid":"9a16b48f-76af-4198-8fe3-d2c20a776a3e","created":"2024-03-23T14:34:48.256967+0000","modified":"2024-03-23T14:37:38.262926+0000","last_up_change":"2024-03-23T14:34:59.592238+0000","last_in_change":"2024-03-23T14:34:51.204801+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-23T14:35:00.521058+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"b5759fd7-481a-4d08-94da-d5f47464dc44","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6820","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6823","nonce":3143302717}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6825","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6827","nonce":3143302717}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6830","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6831","nonce":3143302717}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6828","nonce":3143302717},{"type":"v1","addr":"172.21.15.161:6829","nonce":3143302717}]},"public_addr":"172.21.15.161:6823/3143302717","cluster_addr":"172.21.15.161:6827/3143302717","heartbeat_back_addr":"172.21.15.161:6831/3143302717","heartbeat_front_addr":"172.21.15.161:6829/3143302717","state":["exists","up"]},{"osd":1,"uuid":"6c8004b2-58d7-46c3-a737-d376a959fde1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6808","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6810","nonce":2842040386}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6812","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6813","nonce":2842040386}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6818","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6821","nonce":2842040386}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6815","nonce":2842040386},{"type":"v1","addr":"172.21.15.161:6817","nonce":2842040386}]},"public_addr":"172.21.15.161:6810/2842040386","cluster_addr":"172.21.15.161:6813/2842040386","heartbeat_back_addr":"172.21.15.161:6821/2842040386","heartbeat_front_addr":"172.21.15.161:6817/2842040386","state":["exists","up"]},{"osd":2,"uuid":"cb73706e-1fab-4f91-92c3-91c131ae2240","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6809","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6811","nonce":4063400408}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6814","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6816","nonce":4063400408}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6824","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6826","nonce":4063400408}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6819","nonce":4063400408},{"type":"v1","addr":"172.21.15.161:6822","nonce":4063400408}]},"public_addr":"172.21.15.161:6811/4063400408","cluster_addr":"172.21.15.161:6816/4063400408","heartbeat_back_addr":"172.21.15.161:6826/4063400408","heartbeat_front_addr":"172.21.15.161:6822/4063400408","state":["exists","up"]},{"osd":3,"uuid":"37ac7a35-54f4-43ca-8fe1-cbee435e2de6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6800","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6801","nonce":162554479}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6802","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6803","nonce":162554479}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6806","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6807","nonce":162554479}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6804","nonce":162554479},{"type":"v1","addr":"172.21.15.161:6805","nonce":162554479}]},"public_addr":"172.21.15.161:6801/162554479","cluster_addr":"172.21.15.161:6803/162554479","heartbeat_back_addr":"172.21.15.161:6807/162554479","heartbeat_front_addr":"172.21.15.161:6805/162554479","state":["exists","up"]},{"osd":4,"uuid":"b93ae8d2-0aff-4f0f-a8c4-9f4d73cd7f8a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6820","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6822","nonce":3154566470}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6824","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6826","nonce":3154566470}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6830","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6831","nonce":3154566470}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6828","nonce":3154566470},{"type":"v1","addr":"172.21.15.183:6829","nonce":3154566470}]},"public_addr":"172.21.15.183:6822/3154566470","cluster_addr":"172.21.15.183:6826/3154566470","heartbeat_back_addr":"172.21.15.183:6831/3154566470","heartbeat_front_addr":"172.21.15.183:6829/3154566470","state":["exists","up"]},{"osd":5,"uuid":"47e1e2a8-5abe-4727-9978-f52811fa76bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6816","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6817","nonce":2304513884}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6818","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6819","nonce":2304513884}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6825","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6827","nonce":2304513884}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6821","nonce":2304513884},{"type":"v1","addr":"172.21.15.183:6823","nonce":2304513884}]},"public_addr":"172.21.15.183:6817/2304513884","cluster_addr":"172.21.15.183:6819/2304513884","heartbeat_back_addr":"172.21.15.183:6827/2304513884","heartbeat_front_addr":"172.21.15.183:6823/2304513884","state":["exists","up"]},{"osd":6,"uuid":"b65ec625-df08-474d-92af-bd7ab5d58510","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6800","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6801","nonce":963839040}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6802","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6803","nonce":963839040}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6806","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6807","nonce":963839040}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6804","nonce":963839040},{"type":"v1","addr":"172.21.15.183:6805","nonce":963839040}]},"public_addr":"172.21.15.183:6801/963839040","cluster_addr":"172.21.15.183:6803/963839040","heartbeat_back_addr":"172.21.15.183:6807/963839040","heartbeat_front_addr":"172.21.15.183:6805/963839040","state":["exists","up"]},{"osd":7,"uuid":"0670b2b9-514f-4a9a-9da6-d35c981423dc","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6808","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6809","nonce":4120799670}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6810","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6811","nonce":4120799670}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6814","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6815","nonce":4120799670}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6812","nonce":4120799670},{"type":"v1","addr":"172.21.15.183:6813","nonce":4120799670}]},"public_addr":"172.21.15.183:6809/4120799670","cluster_addr":"172.21.15.183:6811/4120799670","heartbeat_back_addr":"172.21.15.183:6815/4120799670","heartbeat_front_addr":"172.21.15.183:6813/4120799670","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.388491+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.380782+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.421365+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.334422+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.427310+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.338768+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.356852+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-23T14:34:56.337293+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.161:6835/1671955402":"2024-03-24T14:37:35.181374+0000","172.21.15.161:6833/1671955402":"2024-03-24T14:37:35.181374+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-23T14:37:41.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda4006b810 msgr2=0x7fda4006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda4006b810 0x7fda4006dcd0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fda54036000 tx=0x7fda5403c040 comp rx=0 tx=0).stop 2024-03-23T14:37:41.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 msgr2=0x7fda60153e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:41.521 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda60153e40 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7fda5000c3c0 tx=0x7fda5000c880 comp rx=0 tx=0).stop 2024-03-23T14:37:41.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 shutdown_connections 2024-03-23T14:37:41.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fda4006b810 0x7fda4006dcd0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fda600724e0 0x7fda60154380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fda60072100 0x7fda60153e40 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 --2- 172.21.15.161:0/3068007985 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fda6006eca0 0x7fda6014ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:41.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 >> 172.21.15.161:0/3068007985 conn(0x7fda600b17f0 msgr2=0x7fda6015ad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:41.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 shutdown_connections 2024-03-23T14:37:41.522 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:41.525+0000 7fda64993640 1 -- 172.21.15.161:0/3068007985 wait complete. 2024-03-23T14:37:42.533 INFO:tasks.ceph:Scrubbing osd.0 2024-03-23T14:37:42.534 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-03-23T14:37:42.648 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.652+0000 7fb77ceae640 1 Processor -- start 2024-03-23T14:37:42.649 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.652+0000 7fb77ceae640 1 -- start start 2024-03-23T14:37:42.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb77ceae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 0x7fb7780cf1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb77ceae640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb7780cf700 0x7fb7780d2860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb77ceae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7780d1e10 con 0x7fb77815e550 2024-03-23T14:37:42.650 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb77ceae640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb7780d4970 con 0x7fb7780cf700 2024-03-23T14:37:42.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb77ceae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb7780d4c50 con 0x7fb7780cfca0 2024-03-23T14:37:42.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb776d9d640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb7780cf700 0x7fb7780d2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb77659c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb77815e550 0x7fb7780ceab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41022/0 (socket says 172.21.15.161:41022) 2024-03-23T14:37:42.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb77659c640 1 -- 172.21.15.161:0/2102826211 learned_addr learned my addr 172.21.15.161:0/2102826211 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:42.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.653+0000 7fb76ffff640 1 --2- 172.21.15.161:0/2102826211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 0x7fb7780cf1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.654+0000 7fb776d9d640 1 -- 172.21.15.161:0/2102826211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 msgr2=0x7fb7780cf1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:42.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.654+0000 7fb776d9d640 1 --2- 172.21.15.161:0/2102826211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 0x7fb7780cf1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.651 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.654+0000 7fb776d9d640 1 -- 172.21.15.161:0/2102826211 >> v1:172.21.15.161:6789/0 conn(0x7fb77815e550 legacy=0x7fb7780ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.654+0000 7fb775d9b640 1 -- 172.21.15.161:0/2102826211 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515111260 0 0) 0x7fb7780d1e10 con 0x7fb77815e550 2024-03-23T14:37:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.654+0000 7fb776d9d640 1 -- 172.21.15.161:0/2102826211 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7780d2060 con 0x7fb7780cf700 2024-03-23T14:37:42.652 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.654+0000 7fb76ffff640 1 --2- 172.21.15.161:0/2102826211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 0x7fb7780cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:42.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb776d9d640 1 --2- 172.21.15.161:0/2102826211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 0x7fb7780d2860 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb770009d90 tx=0x7fb77002be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77b9e196ba30933 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:42.655 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb775d9b640 1 -- 172.21.15.161:0/2102826211 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77002cbc0 con 0x7fb7780cf700 2024-03-23T14:37:42.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb775d9b640 1 -- 172.21.15.161:0/2102826211 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77002cd20 con 0x7fb7780cf700 2024-03-23T14:37:42.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb775d9b640 1 -- 172.21.15.161:0/2102826211 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770034660 con 0x7fb7780cf700 2024-03-23T14:37:42.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb77ceae640 1 -- 172.21.15.161:0/2102826211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 msgr2=0x7fb7780d2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb77ceae640 1 --2- 172.21.15.161:0/2102826211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 0x7fb7780d2860 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7fb770009d90 tx=0x7fb77002be10 comp rx=0 tx=0).stop 2024-03-23T14:37:42.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb77ceae640 1 -- 172.21.15.161:0/2102826211 shutdown_connections 2024-03-23T14:37:42.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb77ceae640 1 --2- 172.21.15.161:0/2102826211 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 0x7fb7780d2860 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb77ceae640 1 --2- 172.21.15.161:0/2102826211 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 0x7fb7780cf1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.656 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.655+0000 7fb77ceae640 1 -- 172.21.15.161:0/2102826211 >> 172.21.15.161:0/2102826211 conn(0x7fb77809d210 msgr2=0x7fb7780d3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:42.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.656+0000 7fb77ceae640 1 -- 172.21.15.161:0/2102826211 shutdown_connections 2024-03-23T14:37:42.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.656+0000 7fb77ceae640 1 -- 172.21.15.161:0/2102826211 wait complete. 2024-03-23T14:37:42.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.657+0000 7fb77ceae640 1 Processor -- start 2024-03-23T14:37:42.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.657+0000 7fb77ceae640 1 -- start start 2024-03-23T14:37:42.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77ceae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 0x7fb7780cece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.657 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77ceae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 0x7fb7780cd2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77ceae640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb77815e550 0x7fb7780cd810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77ceae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb7780ca190 con 0x7fb7780cf700 2024-03-23T14:37:42.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77ceae640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb7780c9510 con 0x7fb77815e550 2024-03-23T14:37:42.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77ceae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb7780c97f0 con 0x7fb7780cfca0 2024-03-23T14:37:42.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 0x7fb7780cece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 0x7fb7780cece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38066/0 (socket says 172.21.15.161:38066) 2024-03-23T14:37:42.658 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 -- 172.21.15.161:0/748905058 learned_addr learned my addr 172.21.15.161:0/748905058 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:42.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb776d9d640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb77815e550 0x7fb7780cd810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb76ffff640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 0x7fb7780cd2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 -- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 msgr2=0x7fb7780cd2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb7780cfca0 0x7fb7780cd2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 -- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb77815e550 msgr2=0x7fb7780cd810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb77815e550 0x7fb7780cd810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 -- 172.21.15.161:0/748905058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb770009a70 con 0x7fb7780cf700 2024-03-23T14:37:42.659 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77659c640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 0x7fb7780cece0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb76800c430 tx=0x7fb76800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:42.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7680074e0 con 0x7fb7780cf700 2024-03-23T14:37:42.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb768007660 con 0x7fb7780cf700 2024-03-23T14:37:42.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.658+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7780c9ad0 con 0x7fb7780cf700 2024-03-23T14:37:42.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.659+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb768004a20 con 0x7fb7780cf700 2024-03-23T14:37:42.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.659+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7780cdd50 con 0x7fb7780cf700 2024-03-23T14:37:42.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.659+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb7780d4620 con 0x7fb7780cf700 2024-03-23T14:37:42.660 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.660+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb768004b80 con 0x7fb7780cf700 2024-03-23T14:37:42.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.660+0000 7fb76effd640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb75806b810 0x7fb75806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.660+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7fb768089b30 con 0x7fb7780cf700 2024-03-23T14:37:42.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.660+0000 7fb76effd640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7fb7580716d0 0x7fb758073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.662 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.660+0000 7fb76ffff640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb75806b810 0x7fb75806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.660+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 --> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7580741a0 con 0x7fb7580716d0 2024-03-23T14:37:42.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.660+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fb76808e2c0 con 0x7fb7780cf700 2024-03-23T14:37:42.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.661+0000 7fb776d9d640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7fb7580716d0 0x7fb758073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.661+0000 7fb76ffff640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb75806b810 0x7fb75806dcd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb7780b2eb0 tx=0x7fb760008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:42.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.661+0000 7fb776d9d640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7fb7580716d0 0x7fb758073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:42.663 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.662+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 <== osd.0 v2:172.21.15.161:6820/3143302717 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb7580741a0 con 0x7fb7580716d0 2024-03-23T14:37:42.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 --> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fb7780a1f60 con 0x7fb7580716d0 2024-03-23T14:37:42.671 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb76effd640 1 -- 172.21.15.161:0/748905058 <== osd.0 v2:172.21.15.161:6820/3143302717 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fb7780a1f60 con 0x7fb7580716d0 2024-03-23T14:37:42.671 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:42.672 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T14:37:42.672 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:42.673 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7fb7580716d0 msgr2=0x7fb758073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.161:6820/3143302717,v1:172.21.15.161:6823/3143302717] conn(0x7fb7580716d0 0x7fb758073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb75806b810 msgr2=0x7fb75806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb75806b810 0x7fb75806dcd0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb7780b2eb0 tx=0x7fb760008040 comp rx=0 tx=0).stop 2024-03-23T14:37:42.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 msgr2=0x7fb7780cece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 --2- 172.21.15.161:0/748905058 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb7780cf700 0x7fb7780cece0 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7fb76800c430 tx=0x7fb76800c8f0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb776d9d640 1 -- 172.21.15.161:0/748905058 reap_dead start 2024-03-23T14:37:42.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 shutdown_connections 2024-03-23T14:37:42.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 >> 172.21.15.161:0/748905058 conn(0x7fb77809d210 msgr2=0x7fb7780b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:42.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 shutdown_connections 2024-03-23T14:37:42.676 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.675+0000 7fb77ceae640 1 -- 172.21.15.161:0/748905058 wait complete. 2024-03-23T14:37:42.682 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-03-23T14:37:42.786 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.791+0000 7f3a4615e640 1 Processor -- start 2024-03-23T14:37:42.787 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.791+0000 7f3a4615e640 1 -- start start 2024-03-23T14:37:42.788 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.791+0000 7f3a4615e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f3a4015e400 0x7f3a400cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.791+0000 7f3a4615e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3a400d2b40 0x7f3a400b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a4615e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3a400d0460 con 0x7f3a400cf9d0 2024-03-23T14:37:42.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a4615e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f3a400d1050 con 0x7f3a400d2b40 2024-03-23T14:37:42.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a4615e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f3a400d06b0 con 0x7f3a4015e400 2024-03-23T14:37:42.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3ffff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f3a400d2b40 0x7f3a400b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3effd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f3a400cf9d0 0x7f3a400d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41036/0 (socket says 172.21.15.161:41036) 2024-03-23T14:37:42.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3effd640 1 -- 172.21.15.161:0/1485959309 learned_addr learned my addr 172.21.15.161:0/1485959309 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:42.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3f7fe640 1 --2- 172.21.15.161:0/1485959309 >> v2:172.21.15.183:3300/0 conn(0x7f3a4015e400 0x7f3a400cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3ffff640 1 -- 172.21.15.161:0/1485959309 >> v2:172.21.15.183:3300/0 conn(0x7f3a4015e400 msgr2=0x7f3a400cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:42.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3ffff640 1 --2- 172.21.15.161:0/1485959309 >> v2:172.21.15.183:3300/0 conn(0x7f3a4015e400 0x7f3a400cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3ffff640 1 -- 172.21.15.161:0/1485959309 >> v1:172.21.15.161:6789/0 conn(0x7f3a400cf9d0 legacy=0x7f3a400d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3ffff640 1 -- 172.21.15.161:0/1485959309 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a400d0990 con 0x7f3a400d2b40 2024-03-23T14:37:42.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3e7fc640 1 -- 172.21.15.161:0/1485959309 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3164715516 0 0) 0x7f3a400d0460 con 0x7f3a400cf9d0 2024-03-23T14:37:42.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.792+0000 7f3a3f7fe640 1 --2- 172.21.15.161:0/1485959309 >> v2:172.21.15.183:3300/0 conn(0x7f3a4015e400 0x7f3a400cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:42.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.793+0000 7f3a3ffff640 1 --2- 172.21.15.161:0/1485959309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400d2b40 0x7f3a400b0a80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3a4008e230 tx=0x7f3a3802bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=862603c13aeb2def server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:42.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.793+0000 7f3a3e7fc640 1 -- 172.21.15.161:0/1485959309 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a3802cc80 con 0x7f3a400d2b40 2024-03-23T14:37:42.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.793+0000 7f3a3e7fc640 1 -- 172.21.15.161:0/1485959309 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a3802cde0 con 0x7f3a400d2b40 2024-03-23T14:37:42.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.793+0000 7f3a3e7fc640 1 -- 172.21.15.161:0/1485959309 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a38034660 con 0x7f3a400d2b40 2024-03-23T14:37:42.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.794+0000 7f3a4615e640 1 -- 172.21.15.161:0/1485959309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400d2b40 msgr2=0x7f3a400b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.794+0000 7f3a4615e640 1 --2- 172.21.15.161:0/1485959309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400d2b40 0x7f3a400b0a80 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f3a4008e230 tx=0x7f3a3802bed0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.794+0000 7f3a4615e640 1 -- 172.21.15.161:0/1485959309 shutdown_connections 2024-03-23T14:37:42.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.794+0000 7f3a4615e640 1 --2- 172.21.15.161:0/1485959309 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400d2b40 0x7f3a400b0a80 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.794+0000 7f3a4615e640 1 --2- 172.21.15.161:0/1485959309 >> v2:172.21.15.183:3300/0 conn(0x7f3a4015e400 0x7f3a400cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.794+0000 7f3a4615e640 1 -- 172.21.15.161:0/1485959309 >> 172.21.15.161:0/1485959309 conn(0x7f3a4009d190 msgr2=0x7f3a400d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:42.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.794+0000 7f3a4615e640 1 -- 172.21.15.161:0/1485959309 shutdown_connections 2024-03-23T14:37:42.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.794+0000 7f3a4615e640 1 -- 172.21.15.161:0/1485959309 wait complete. 2024-03-23T14:37:42.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.795+0000 7f3a4615e640 1 Processor -- start 2024-03-23T14:37:42.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.795+0000 7f3a4615e640 1 -- start start 2024-03-23T14:37:42.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a4615e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400cf9d0 0x7f3a40158f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a4615e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a400d2b40 0x7f3a401f7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a4615e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a4015e400 0x7f3a4015f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a4615e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401f8010 con 0x7f3a400cf9d0 2024-03-23T14:37:42.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a4615e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f3a401f7a50 con 0x7f3a4015e400 2024-03-23T14:37:42.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a4615e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f3a401f7d30 con 0x7f3a400d2b40 2024-03-23T14:37:42.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400cf9d0 0x7f3a40158f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400cf9d0 0x7f3a40158f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38090/0 (socket says 172.21.15.161:38090) 2024-03-23T14:37:42.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 -- 172.21.15.161:0/4057886650 learned_addr learned my addr 172.21.15.161:0/4057886650 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:42.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3ffff640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a4015e400 0x7f3a4015f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3effd640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a400d2b40 0x7f3a401f7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 -- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a400d2b40 msgr2=0x7f3a401f7100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a400d2b40 0x7f3a401f7100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 -- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a4015e400 msgr2=0x7f3a4015f8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a4015e400 0x7f3a4015f8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 -- 172.21.15.161:0/4057886650 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a38009a70 con 0x7f3a400cf9d0 2024-03-23T14:37:42.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3effd640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a400d2b40 0x7f3a401f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:42.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.796+0000 7f3a3f7fe640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400cf9d0 0x7f3a40158f80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000d7d0 tx=0x7f3a3000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:42.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.797+0000 7f3a27fff640 1 -- 172.21.15.161:0/4057886650 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30004030 con 0x7f3a400cf9d0 2024-03-23T14:37:42.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.797+0000 7f3a27fff640 1 -- 172.21.15.161:0/4057886650 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a30004190 con 0x7f3a400cf9d0 2024-03-23T14:37:42.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.797+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a400c7cc0 con 0x7f3a400cf9d0 2024-03-23T14:37:42.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.797+0000 7f3a27fff640 1 -- 172.21.15.161:0/4057886650 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a30004520 con 0x7f3a400cf9d0 2024-03-23T14:37:42.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.797+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a401594c0 con 0x7f3a400cf9d0 2024-03-23T14:37:42.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.798+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a400a1e00 con 0x7f3a400cf9d0 2024-03-23T14:37:42.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.798+0000 7f3a27fff640 1 -- 172.21.15.161:0/4057886650 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f3a300096a0 con 0x7f3a400cf9d0 2024-03-23T14:37:42.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.798+0000 7f3a27fff640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a2c06b810 0x7f3a2c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:42.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.799+0000 7f3a27fff640 1 -- 172.21.15.161:0/4057886650 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f3a3008a070 con 0x7f3a400cf9d0 2024-03-23T14:37:42.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.799+0000 7f3a3effd640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a2c06b810 0x7f3a2c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:42.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.799+0000 7f3a3effd640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a2c06b810 0x7f3a2c06dcd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3a400c7810 tx=0x7f3a34009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:42.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.803+0000 7f3a27fff640 1 -- 172.21.15.161:0/4057886650 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f3a30054d90 con 0x7f3a400cf9d0 2024-03-23T14:37:42.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.982+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f3a400d1470 con 0x7f3a2c06b810 2024-03-23T14:37:42.978 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.983+0000 7f3a27fff640 1 -- 172.21.15.161:0/4057886650 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f3a340013d0 con 0x7f3a2c06b810 2024-03-23T14:37:42.979 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 0 to deep-scrub 2024-03-23T14:37:42.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a2c06b810 msgr2=0x7f3a2c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a2c06b810 0x7f3a2c06dcd0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f3a400c7810 tx=0x7f3a34009040 comp rx=0 tx=0).stop 2024-03-23T14:37:42.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400cf9d0 msgr2=0x7f3a40158f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:42.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400cf9d0 0x7f3a40158f80 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f3a3000d7d0 tx=0x7f3a3000dc90 comp rx=0 tx=0).stop 2024-03-23T14:37:42.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 shutdown_connections 2024-03-23T14:37:42.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f3a2c06b810 0x7f3a2c06dcd0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f3a4015e400 0x7f3a4015f8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f3a400d2b40 0x7f3a401f7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 --2- 172.21.15.161:0/4057886650 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f3a400cf9d0 0x7f3a40158f80 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:42.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 >> 172.21.15.161:0/4057886650 conn(0x7f3a4009d190 msgr2=0x7f3a400b1d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:42.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 shutdown_connections 2024-03-23T14:37:42.983 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:42.985+0000 7f3a4615e640 1 -- 172.21.15.161:0/4057886650 wait complete. 2024-03-23T14:37:42.992 INFO:tasks.ceph:Scrubbing osd.1 2024-03-23T14:37:42.992 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-03-23T14:37:43.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.102+0000 7efd8aeae640 1 Processor -- start 2024-03-23T14:37:43.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.102+0000 7efd8aeae640 1 -- start start 2024-03-23T14:37:43.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd8aeae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8406fe70 0x7efd840cfb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd8aeae640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd840cd500 con 0x7efd840cd990 2024-03-23T14:37:43.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd8aeae640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd840ccbd0 con 0x7efd8415c080 2024-03-23T14:37:43.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd8aeae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efd840cddf0 con 0x7efd8406fe70 2024-03-23T14:37:43.102 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd83fff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efd840cd990 0x7efd8406f760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41040/0 (socket says 172.21.15.161:41040) 2024-03-23T14:37:43.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd83fff640 1 -- 172.21.15.161:0/3180616759 learned_addr learned my addr 172.21.15.161:0/3180616759 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:43.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1282400274 0 0) 0x7efd840cd500 con 0x7efd840cd990 2024-03-23T14:37:43.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd8944b640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8406fe70 0x7efd840cfb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd74003520 con 0x7efd840cd990 2024-03-23T14:37:43.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4139925653 0 0) 0x7efd74003520 con 0x7efd840cd990 2024-03-23T14:37:43.103 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8406fe70 msgr2=0x7efd840cfb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8406fe70 0x7efd840cfb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 >> v1:172.21.15.183:6789/0 conn(0x7efd8415c080 legacy=0x7efd840cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd840cceb0 con 0x7efd840cd990 2024-03-23T14:37:43.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7efd70001670 con 0x7efd840cd990 2024-03-23T14:37:43.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 >> v1:172.21.15.161:6789/0 conn(0x7efd840cd990 legacy=0x7efd8406f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd740039a0 0x7efd74003dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd74004c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8406fe70 0x7efd74005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd7400ba30 con 0x7efd740047c0 2024-03-23T14:37:43.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efd7400bba0 con 0x7efd740039a0 2024-03-23T14:37:43.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efd7400bd60 con 0x7efd8406fe70 2024-03-23T14:37:43.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.103+0000 7efd83fff640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd74004c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.104+0000 7efd88c4a640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd740039a0 0x7efd74003dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.104+0000 7efd8944b640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8406fe70 0x7efd74005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.104+0000 7efd83fff640 1 -- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8406fe70 msgr2=0x7efd74005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.104+0000 7efd83fff640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8406fe70 0x7efd74005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.104+0000 7efd83fff640 1 -- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd740039a0 msgr2=0x7efd74003dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.104+0000 7efd83fff640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd740039a0 0x7efd74003dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.104+0000 7efd83fff640 1 -- 172.21.15.161:0/3180616759 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd74003520 con 0x7efd740047c0 2024-03-23T14:37:43.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.104+0000 7efd83fff640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd74004c40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7efd7000c210 tx=0x7efd740057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=439d2f6dfab64bc0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd70005ad0 con 0x7efd740047c0 2024-03-23T14:37:43.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd70005c30 con 0x7efd740047c0 2024-03-23T14:37:43.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd837fe640 1 -- 172.21.15.161:0/3180616759 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7002c870 con 0x7efd740047c0 2024-03-23T14:37:43.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd8aeae640 1 -- 172.21.15.161:0/3180616759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 msgr2=0x7efd74004c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd8aeae640 1 --2- 172.21.15.161:0/3180616759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd74004c40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7efd7000c210 tx=0x7efd740057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd8944b640 1 -- 172.21.15.161:0/3180616759 reap_dead start 2024-03-23T14:37:43.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd8aeae640 1 -- 172.21.15.161:0/3180616759 shutdown_connections 2024-03-23T14:37:43.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd8aeae640 1 -- 172.21.15.161:0/3180616759 >> 172.21.15.161:0/3180616759 conn(0x7efd840d1c70 msgr2=0x7efd840d2080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd8aeae640 1 -- 172.21.15.161:0/3180616759 shutdown_connections 2024-03-23T14:37:43.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.105+0000 7efd8aeae640 1 -- 172.21.15.161:0/3180616759 wait complete. 2024-03-23T14:37:43.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.106+0000 7efd8aeae640 1 Processor -- start 2024-03-23T14:37:43.108 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.106+0000 7efd8aeae640 1 -- start start 2024-03-23T14:37:43.111 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd8aeae640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd840cbb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd8aeae640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd841f4f60 0x7efd8414ba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd8aeae640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8415a0d0 0x7efd8415a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd8aeae640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efd840c8d40 con 0x7efd740047c0 2024-03-23T14:37:43.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd8aeae640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efd8415aa90 con 0x7efd841f4f60 2024-03-23T14:37:43.112 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd8aeae640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efd8414bf50 con 0x7efd8415a0d0 2024-03-23T14:37:43.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd88c4a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd840cbb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd88c4a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd840cbb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38110/0 (socket says 172.21.15.161:38110) 2024-03-23T14:37:43.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.107+0000 7efd88c4a640 1 -- 172.21.15.161:0/1398997136 learned_addr learned my addr 172.21.15.161:0/1398997136 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:43.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd88c4a640 1 -- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8415a0d0 msgr2=0x7efd8415a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd88c4a640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd8415a0d0 0x7efd8415a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd88c4a640 1 -- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd841f4f60 msgr2=0x7efd8414ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.113 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd88c4a640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd841f4f60 0x7efd8414ba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd88c4a640 1 -- 172.21.15.161:0/1398997136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd841f53e0 con 0x7efd740047c0 2024-03-23T14:37:43.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd88c4a640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd840cbb10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efd7800f570 tx=0x7efd7800fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd780052e0 con 0x7efd740047c0 2024-03-23T14:37:43.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd78005440 con 0x7efd740047c0 2024-03-23T14:37:43.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd841f56b0 con 0x7efd740047c0 2024-03-23T14:37:43.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd78003810 con 0x7efd740047c0 2024-03-23T14:37:43.114 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.108+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd841f5c70 con 0x7efd740047c0 2024-03-23T14:37:43.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.109+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efd840cc880 con 0x7efd740047c0 2024-03-23T14:37:43.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.110+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efd78003970 con 0x7efd740047c0 2024-03-23T14:37:43.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.110+0000 7efd817fa640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd6806b810 0x7efd6806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.110+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7efd78089d00 con 0x7efd740047c0 2024-03-23T14:37:43.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.110+0000 7efd817fa640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7efd680716d0 0x7efd68073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.110+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 --> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd680741a0 con 0x7efd680716d0 2024-03-23T14:37:43.115 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.110+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7efd7808c3c0 con 0x7efd740047c0 2024-03-23T14:37:43.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.110+0000 7efd83fff640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd6806b810 0x7efd6806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.110+0000 7efd8944b640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7efd680716d0 0x7efd68073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.111+0000 7efd8944b640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7efd680716d0 0x7efd68073af0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.111+0000 7efd83fff640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd6806b810 0x7efd6806dcd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efd70034000 tx=0x7efd70007000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.116 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.111+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 <== osd.1 v2:172.21.15.161:6808/2842040386 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efd680741a0 con 0x7efd680716d0 2024-03-23T14:37:43.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.124+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 --> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7efd840c90d0 con 0x7efd680716d0 2024-03-23T14:37:43.120 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.124+0000 7efd817fa640 1 -- 172.21.15.161:0/1398997136 <== osd.1 v2:172.21.15.161:6808/2842040386 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7efd840c90d0 con 0x7efd680716d0 2024-03-23T14:37:43.121 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:43.121 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T14:37:43.121 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:43.122 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7efd680716d0 msgr2=0x7efd68073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.161:6808/2842040386,v1:172.21.15.161:6810/2842040386] conn(0x7efd680716d0 0x7efd68073af0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd6806b810 msgr2=0x7efd6806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd6806b810 0x7efd6806dcd0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efd70034000 tx=0x7efd70007000 comp rx=0 tx=0).stop 2024-03-23T14:37:43.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 msgr2=0x7efd840cbb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 --2- 172.21.15.161:0/1398997136 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd740047c0 0x7efd840cbb10 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7efd7800f570 tx=0x7efd7800fa30 comp rx=0 tx=0).stop 2024-03-23T14:37:43.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8944b640 1 -- 172.21.15.161:0/1398997136 reap_dead start 2024-03-23T14:37:43.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 shutdown_connections 2024-03-23T14:37:43.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 >> 172.21.15.161:0/1398997136 conn(0x7efd840d1c70 msgr2=0x7efd8415d720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 shutdown_connections 2024-03-23T14:37:43.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.125+0000 7efd8aeae640 1 -- 172.21.15.161:0/1398997136 wait complete. 2024-03-23T14:37:43.132 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-03-23T14:37:43.251 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.241+0000 7f13e7993640 1 Processor -- start 2024-03-23T14:37:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.241+0000 7f13e7993640 1 -- start start 2024-03-23T14:37:43.252 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e7993640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e015e400 0x7f13e00cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e7993640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f13e00d2b40 0x7f13e00b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.253 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e7993640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e00d0460 con 0x7f13e00cf9d0 2024-03-23T14:37:43.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e7993640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f13e00d1050 con 0x7f13e00d2b40 2024-03-23T14:37:43.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e7993640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f13e00d06b0 con 0x7f13e015e400 2024-03-23T14:37:43.254 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e5f30640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f13e00d2b40 0x7f13e00b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e5f30640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f13e00d2b40 0x7f13e00b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38122/0 (socket says 172.21.15.161:38122) 2024-03-23T14:37:43.255 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e5f30640 1 -- 172.21.15.161:0/2561356355 learned_addr learned my addr 172.21.15.161:0/2561356355 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e572f640 1 --2- 172.21.15.161:0/2561356355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e015e400 0x7f13e00cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e5f30640 1 -- 172.21.15.161:0/2561356355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e015e400 msgr2=0x7f13e00cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.256 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e5f30640 1 --2- 172.21.15.161:0/2561356355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e015e400 0x7f13e00cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e5f30640 1 -- 172.21.15.161:0/2561356355 >> v1:172.21.15.183:6789/0 conn(0x7f13e00cf9d0 legacy=0x7f13e00d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.257 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.242+0000 7f13e5f30640 1 -- 172.21.15.161:0/2561356355 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e00d0990 con 0x7f13e00d2b40 2024-03-23T14:37:43.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.243+0000 7f13e5f30640 1 --2- 172.21.15.161:0/2561356355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00d2b40 0x7f13e00b0a80 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009870 tx=0x7f13dc02b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf7ab4a043e9bd35 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.243+0000 7f13d7fff640 1 -- 172.21.15.161:0/2561356355 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc030840 con 0x7f13e00d2b40 2024-03-23T14:37:43.258 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.243+0000 7f13d7fff640 1 -- 172.21.15.161:0/2561356355 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13dc0309a0 con 0x7f13e00d2b40 2024-03-23T14:37:43.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.243+0000 7f13d7fff640 1 -- 172.21.15.161:0/2561356355 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13dc030d30 con 0x7f13e00d2b40 2024-03-23T14:37:43.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.244+0000 7f13e7993640 1 -- 172.21.15.161:0/2561356355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00d2b40 msgr2=0x7f13e00b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.259 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.244+0000 7f13e7993640 1 --2- 172.21.15.161:0/2561356355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00d2b40 0x7f13e00b0a80 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009870 tx=0x7f13dc02b980 comp rx=0 tx=0).stop 2024-03-23T14:37:43.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.244+0000 7f13e7993640 1 -- 172.21.15.161:0/2561356355 shutdown_connections 2024-03-23T14:37:43.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.244+0000 7f13e7993640 1 --2- 172.21.15.161:0/2561356355 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00d2b40 0x7f13e00b0a80 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.244+0000 7f13e7993640 1 --2- 172.21.15.161:0/2561356355 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e015e400 0x7f13e00cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.260 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.244+0000 7f13e7993640 1 -- 172.21.15.161:0/2561356355 >> 172.21.15.161:0/2561356355 conn(0x7f13e009d190 msgr2=0x7f13e00d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.244+0000 7f13e7993640 1 -- 172.21.15.161:0/2561356355 shutdown_connections 2024-03-23T14:37:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.244+0000 7f13e7993640 1 -- 172.21.15.161:0/2561356355 wait complete. 2024-03-23T14:37:43.261 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.246+0000 7f13e7993640 1 Processor -- start 2024-03-23T14:37:43.262 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.246+0000 7f13e7993640 1 -- start start 2024-03-23T14:37:43.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e7993640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00cf9d0 0x7f13e00b8790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e7993640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e00d2b40 0x7f13e01f4f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.267 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e7993640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e015e400 0x7f13e00ca320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e7993640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f13e00b9110 con 0x7f13e00cf9d0 2024-03-23T14:37:43.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e7993640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f13e00c8170 con 0x7f13e015e400 2024-03-23T14:37:43.268 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e7993640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f13e00b8e30 con 0x7f13e00d2b40 2024-03-23T14:37:43.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e572f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00cf9d0 0x7f13e00b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e572f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00cf9d0 0x7f13e00b8790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38124/0 (socket says 172.21.15.161:38124) 2024-03-23T14:37:43.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e572f640 1 -- 172.21.15.161:0/783531447 learned_addr learned my addr 172.21.15.161:0/783531447 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:43.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e5f30640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e015e400 0x7f13e00ca320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e4f2e640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e00d2b40 0x7f13e01f4f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.269 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e572f640 1 -- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e00d2b40 msgr2=0x7f13e01f4f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e572f640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e00d2b40 0x7f13e01f4f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e572f640 1 -- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e015e400 msgr2=0x7f13e00ca320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.270 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e572f640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e015e400 0x7f13e00ca320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e572f640 1 -- 172.21.15.161:0/783531447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13dc0094d0 con 0x7f13e00cf9d0 2024-03-23T14:37:43.271 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.247+0000 7f13e5f30640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e015e400 0x7f13e00ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:43.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.248+0000 7f13e572f640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00cf9d0 0x7f13e00b8790 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f13d8007e40 tx=0x7f13d800e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.272 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.248+0000 7f13d5ffb640 1 -- 172.21.15.161:0/783531447 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8014620 con 0x7f13e00cf9d0 2024-03-23T14:37:43.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.248+0000 7f13d5ffb640 1 -- 172.21.15.161:0/783531447 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f13d800ee90 con 0x7f13e00cf9d0 2024-03-23T14:37:43.273 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.248+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e00c7b20 con 0x7f13e00cf9d0 2024-03-23T14:37:43.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.248+0000 7f13d5ffb640 1 -- 172.21.15.161:0/783531447 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80149c0 con 0x7f13e00cf9d0 2024-03-23T14:37:43.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.248+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e00b5b00 con 0x7f13e00cf9d0 2024-03-23T14:37:43.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.249+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e00c7980 con 0x7f13e00cf9d0 2024-03-23T14:37:43.274 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.249+0000 7f13d5ffb640 1 -- 172.21.15.161:0/783531447 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f13d8014b20 con 0x7f13e00cf9d0 2024-03-23T14:37:43.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.249+0000 7f13d5ffb640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13c406b810 0x7f13c406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.250+0000 7f13d5ffb640 1 -- 172.21.15.161:0/783531447 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f13d808a7a0 con 0x7f13e00cf9d0 2024-03-23T14:37:43.275 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.250+0000 7f13e4f2e640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13c406b810 0x7f13c406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.250+0000 7f13e4f2e640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13c406b810 0x7f13c406dcd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f13e01f4f20 tx=0x7f13d0009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.276 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.254+0000 7f13d5ffb640 1 -- 172.21.15.161:0/783531447 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f13e00c7980 con 0x7f13e00cf9d0 2024-03-23T14:37:43.423 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.427+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f13e00d1470 con 0x7f13c406b810 2024-03-23T14:37:43.424 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.428+0000 7f13d5ffb640 1 -- 172.21.15.161:0/783531447 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f13d00050d0 con 0x7f13c406b810 2024-03-23T14:37:43.424 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 1 to deep-scrub 2024-03-23T14:37:43.428 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13c406b810 msgr2=0x7f13c406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13c406b810 0x7f13c406dcd0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f13e01f4f20 tx=0x7f13d0009040 comp rx=0 tx=0).stop 2024-03-23T14:37:43.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00cf9d0 msgr2=0x7f13e00b8790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.429 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00cf9d0 0x7f13e00b8790 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f13d8007e40 tx=0x7f13d800e4b0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 shutdown_connections 2024-03-23T14:37:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f13c406b810 0x7f13c406dcd0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f13e015e400 0x7f13e00ca320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f13e00d2b40 0x7f13e01f4f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.430 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 --2- 172.21.15.161:0/783531447 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f13e00cf9d0 0x7f13e00b8790 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 >> 172.21.15.161:0/783531447 conn(0x7f13e009d190 msgr2=0x7f13e00b1860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 shutdown_connections 2024-03-23T14:37:43.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.430+0000 7f13e7993640 1 -- 172.21.15.161:0/783531447 wait complete. 2024-03-23T14:37:43.437 INFO:tasks.ceph:Scrubbing osd.2 2024-03-23T14:37:43.437 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-03-23T14:37:43.543 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.547+0000 7f33ca7c7640 1 Processor -- start 2024-03-23T14:37:43.544 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.547+0000 7f33ca7c7640 1 -- start start 2024-03-23T14:37:43.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33ca7c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c4144aa0 0x7f33c4070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.552 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33ca7c7640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f33c40710d0 0x7f33c40cfa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33ca7c7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f33c40cff90 0x7f33c40ceca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33ca7c7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f33c40d0420 con 0x7f33c40cff90 2024-03-23T14:37:43.553 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33ca7c7640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f33c4144e80 con 0x7f33c40710d0 2024-03-23T14:37:43.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33ca7c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f33c40b3800 con 0x7f33c4144aa0 2024-03-23T14:37:43.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33c8d64640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f33c40cff90 0x7f33c40ceca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33c8d64640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f33c40cff90 0x7f33c40ceca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38130/0 (socket says 172.21.15.161:38130) 2024-03-23T14:37:43.554 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.548+0000 7f33c8d64640 1 -- 172.21.15.161:0/3665631837 learned_addr learned my addr 172.21.15.161:0/3665631837 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c3fff640 1 --2- 172.21.15.161:0/3665631837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c4144aa0 0x7f33c4070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c37fe640 1 --2- 172.21.15.161:0/3665631837 >> v2:172.21.15.183:3300/0 conn(0x7f33c40710d0 0x7f33c40cfa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.555 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c8d64640 1 -- 172.21.15.161:0/3665631837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c4144aa0 msgr2=0x7f33c4070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c8d64640 1 --2- 172.21.15.161:0/3665631837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c4144aa0 0x7f33c4070b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c8d64640 1 -- 172.21.15.161:0/3665631837 >> v2:172.21.15.183:3300/0 conn(0x7f33c40710d0 msgr2=0x7f33c40cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c8d64640 1 --2- 172.21.15.161:0/3665631837 >> v2:172.21.15.183:3300/0 conn(0x7f33c40710d0 0x7f33c40cfa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.556 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c8d64640 1 -- 172.21.15.161:0/3665631837 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33c40b3ae0 con 0x7f33c40cff90 2024-03-23T14:37:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c37fe640 1 --2- 172.21.15.161:0/3665631837 >> v2:172.21.15.183:3300/0 conn(0x7f33c40710d0 0x7f33c40cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.549+0000 7f33c8d64640 1 --2- 172.21.15.161:0/3665631837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c40cff90 0x7f33c40ceca0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f33c406ef10 tx=0x7f33bc02bdb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f184c4a64105d581 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33c2ffd640 1 -- 172.21.15.161:0/3665631837 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc02c950 con 0x7f33c40cff90 2024-03-23T14:37:43.557 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33c2ffd640 1 -- 172.21.15.161:0/3665631837 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33bc02cab0 con 0x7f33c40cff90 2024-03-23T14:37:43.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33c2ffd640 1 -- 172.21.15.161:0/3665631837 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33bc02ce40 con 0x7f33c40cff90 2024-03-23T14:37:43.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3665631837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c40cff90 msgr2=0x7f33c40ceca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 --2- 172.21.15.161:0/3665631837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c40cff90 0x7f33c40ceca0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7f33c406ef10 tx=0x7f33bc02bdb0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3665631837 shutdown_connections 2024-03-23T14:37:43.558 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 --2- 172.21.15.161:0/3665631837 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c40cff90 0x7f33c40ceca0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 --2- 172.21.15.161:0/3665631837 >> v2:172.21.15.183:3300/0 conn(0x7f33c40710d0 0x7f33c40cfa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 --2- 172.21.15.161:0/3665631837 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c4144aa0 0x7f33c4070b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.559 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3665631837 >> 172.21.15.161:0/3665631837 conn(0x7f33c406b080 msgr2=0x7f33c406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3665631837 shutdown_connections 2024-03-23T14:37:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.550+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3665631837 wait complete. 2024-03-23T14:37:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.551+0000 7f33ca7c7640 1 Processor -- start 2024-03-23T14:37:43.560 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.552+0000 7f33ca7c7640 1 -- start start 2024-03-23T14:37:43.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.552+0000 7f33ca7c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c40cff90 0x7f33c406dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.552+0000 7f33ca7c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c4144aa0 0x7f33c406e280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.561 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.552+0000 7f33ca7c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33c40c4a60 0x7f33c40c4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.552+0000 7f33ca7c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41f40d0 con 0x7f33c4144aa0 2024-03-23T14:37:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.552+0000 7f33ca7c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f33c41f43b0 con 0x7f33c40c4a60 2024-03-23T14:37:43.562 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.552+0000 7f33ca7c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f33c41f4690 con 0x7f33c40cff90 2024-03-23T14:37:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.552+0000 7f33c37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c4144aa0 0x7f33c406e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c37fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c4144aa0 0x7f33c406e280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38140/0 (socket says 172.21.15.161:38140) 2024-03-23T14:37:43.563 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c37fe640 1 -- 172.21.15.161:0/3674612078 learned_addr learned my addr 172.21.15.161:0/3674612078 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:43.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c3fff640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c40cff90 0x7f33c406dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c8d64640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33c40c4a60 0x7f33c40c4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c37fe640 1 -- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c40cff90 msgr2=0x7f33c406dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c37fe640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f33c40cff90 0x7f33c406dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c37fe640 1 -- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33c40c4a60 msgr2=0x7f33c40c4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c37fe640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f33c40c4a60 0x7f33c40c4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c37fe640 1 -- 172.21.15.161:0/3674612078 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33bc009b30 con 0x7f33c4144aa0 2024-03-23T14:37:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c37fe640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c4144aa0 0x7f33c406e280 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f33b400e730 tx=0x7f33b400ebf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b4004230 con 0x7f33c4144aa0 2024-03-23T14:37:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33b4004390 con 0x7f33c4144aa0 2024-03-23T14:37:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33c41f4970 con 0x7f33c4144aa0 2024-03-23T14:37:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33b400c8f0 con 0x7f33c4144aa0 2024-03-23T14:37:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.553+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33c41f3d40 con 0x7f33c4144aa0 2024-03-23T14:37:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.554+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f33c40ce780 con 0x7f33c4144aa0 2024-03-23T14:37:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.555+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f33b400ca50 con 0x7f33c4144aa0 2024-03-23T14:37:43.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.555+0000 7f33c0ff9640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f33b006b810 0x7f33b006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.555+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f33b4089a40 con 0x7f33c4144aa0 2024-03-23T14:37:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.555+0000 7f33c0ff9640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f33b00716d0 0x7f33b0073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.555+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 --> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33b00741a0 con 0x7f33b00716d0 2024-03-23T14:37:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.555+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f33b408e2c0 con 0x7f33c4144aa0 2024-03-23T14:37:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.555+0000 7f33c3fff640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f33b006b810 0x7f33b006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.555+0000 7f33c8d64640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f33b00716d0 0x7f33b0073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.556+0000 7f33c8d64640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f33b00716d0 0x7f33b0073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.556+0000 7f33c3fff640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f33b006b810 0x7f33b006dcd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f33b8007920 tx=0x7f33b8008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.556+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 <== osd.2 v2:172.21.15.161:6809/4063400408 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f33b00741a0 con 0x7f33b00716d0 2024-03-23T14:37:43.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.569+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 --> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f33c40735e0 con 0x7f33b00716d0 2024-03-23T14:37:43.569 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.569+0000 7f33c0ff9640 1 -- 172.21.15.161:0/3674612078 <== osd.2 v2:172.21.15.161:6809/4063400408 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f33c40735e0 con 0x7f33b00716d0 2024-03-23T14:37:43.569 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:43.570 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T14:37:43.570 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:43.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f33b00716d0 msgr2=0x7f33b0073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.161:6809/4063400408,v1:172.21.15.161:6811/4063400408] conn(0x7f33b00716d0 0x7f33b0073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f33b006b810 msgr2=0x7f33b006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f33b006b810 0x7f33b006dcd0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f33b8007920 tx=0x7f33b8008040 comp rx=0 tx=0).stop 2024-03-23T14:37:43.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c4144aa0 msgr2=0x7f33c406e280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 --2- 172.21.15.161:0/3674612078 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f33c4144aa0 0x7f33c406e280 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7f33b400e730 tx=0x7f33b400ebf0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33c8d64640 1 -- 172.21.15.161:0/3674612078 reap_dead start 2024-03-23T14:37:43.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 shutdown_connections 2024-03-23T14:37:43.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 >> 172.21.15.161:0/3674612078 conn(0x7f33c406b080 msgr2=0x7f33c4142b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 shutdown_connections 2024-03-23T14:37:43.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.570+0000 7f33ca7c7640 1 -- 172.21.15.161:0/3674612078 wait complete. 2024-03-23T14:37:43.576 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-03-23T14:37:43.690 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.686+0000 7f64cad3e640 1 Processor -- start 2024-03-23T14:37:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.686+0000 7f64cad3e640 1 -- start start 2024-03-23T14:37:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64cad3e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f64c4162970 0x7f64c415ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64cad3e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f64c40d2a70 0x7f64c415e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.691 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64cad3e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c40d2270 con 0x7f64c40d2690 2024-03-23T14:37:43.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64cad3e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f64c40d06a0 con 0x7f64c40d2a70 2024-03-23T14:37:43.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64cad3e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f64c40d0980 con 0x7f64c4162970 2024-03-23T14:37:43.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c92db640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f64c40d2a70 0x7f64c415e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c92db640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f64c40d2a70 0x7f64c415e330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38152/0 (socket says 172.21.15.161:38152) 2024-03-23T14:37:43.692 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c8ada640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f64c40d2690 0x7f64c40d1b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41056/0 (socket says 172.21.15.161:41056) 2024-03-23T14:37:43.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c92db640 1 -- 172.21.15.161:0/235948065 learned_addr learned my addr 172.21.15.161:0/235948065 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:43.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c3fff640 1 --2- 172.21.15.161:0/235948065 >> v2:172.21.15.183:3300/0 conn(0x7f64c4162970 0x7f64c415ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c92db640 1 -- 172.21.15.161:0/235948065 >> v2:172.21.15.183:3300/0 conn(0x7f64c4162970 msgr2=0x7f64c415ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c92db640 1 --2- 172.21.15.161:0/235948065 >> v2:172.21.15.183:3300/0 conn(0x7f64c4162970 0x7f64c415ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.693 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c92db640 1 -- 172.21.15.161:0/235948065 >> v1:172.21.15.161:6789/0 conn(0x7f64c40d2690 legacy=0x7f64c40d1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c92db640 1 -- 172.21.15.161:0/235948065 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c40d0c60 con 0x7f64c40d2a70 2024-03-23T14:37:43.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.687+0000 7f64c37fe640 1 -- 172.21.15.161:0/235948065 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 303639488 0 0) 0x7f64c40d2270 con 0x7f64c40d2690 2024-03-23T14:37:43.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.688+0000 7f64c92db640 1 --2- 172.21.15.161:0/235948065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c415e330 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f64c408c4c0 tx=0x7f64bc02bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=736dc3d608f2dc81 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.694 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.688+0000 7f64c37fe640 1 -- 172.21.15.161:0/235948065 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc02cb20 con 0x7f64c40d2a70 2024-03-23T14:37:43.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.688+0000 7f64c37fe640 1 -- 172.21.15.161:0/235948065 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64bc02cc80 con 0x7f64c40d2a70 2024-03-23T14:37:43.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.688+0000 7f64c37fe640 1 -- 172.21.15.161:0/235948065 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc0343f0 con 0x7f64c40d2a70 2024-03-23T14:37:43.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.689+0000 7f64cad3e640 1 -- 172.21.15.161:0/235948065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 msgr2=0x7f64c415e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.689+0000 7f64cad3e640 1 --2- 172.21.15.161:0/235948065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c415e330 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f64c408c4c0 tx=0x7f64bc02bf80 comp rx=0 tx=0).stop 2024-03-23T14:37:43.695 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.689+0000 7f64cad3e640 1 -- 172.21.15.161:0/235948065 shutdown_connections 2024-03-23T14:37:43.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.689+0000 7f64cad3e640 1 --2- 172.21.15.161:0/235948065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c415e330 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.689+0000 7f64cad3e640 1 --2- 172.21.15.161:0/235948065 >> v2:172.21.15.183:3300/0 conn(0x7f64c4162970 0x7f64c415ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.689+0000 7f64cad3e640 1 -- 172.21.15.161:0/235948065 >> 172.21.15.161:0/235948065 conn(0x7f64c409d1b0 msgr2=0x7f64c40d3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.689+0000 7f64cad3e640 1 -- 172.21.15.161:0/235948065 shutdown_connections 2024-03-23T14:37:43.696 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.689+0000 7f64cad3e640 1 -- 172.21.15.161:0/235948065 wait complete. 2024-03-23T14:37:43.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.690+0000 7f64cad3e640 1 Processor -- start 2024-03-23T14:37:43.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.690+0000 7f64cad3e640 1 -- start start 2024-03-23T14:37:43.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64cad3e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64c40d2690 0x7f64c4152560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64cad3e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c40c9770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.697 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64cad3e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64c4162970 0x7f64c40b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64cad3e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f64c40b6a60 con 0x7f64c40d2a70 2024-03-23T14:37:43.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64cad3e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f64c40b6d40 con 0x7f64c40d2690 2024-03-23T14:37:43.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64cad3e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f64c40b7020 con 0x7f64c4162970 2024-03-23T14:37:43.698 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c40c9770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c40c9770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38156/0 (socket says 172.21.15.161:38156) 2024-03-23T14:37:43.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c3fff640 1 -- 172.21.15.161:0/4226015459 learned_addr learned my addr 172.21.15.161:0/4226015459 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:43.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c8ada640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64c40d2690 0x7f64c4152560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c92db640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64c4162970 0x7f64c40b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.699 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c3fff640 1 -- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64c4162970 msgr2=0x7f64c40b2030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c3fff640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64c4162970 0x7f64c40b2030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c3fff640 1 -- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64c40d2690 msgr2=0x7f64c4152560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c3fff640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64c40d2690 0x7f64c4152560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.691+0000 7f64c3fff640 1 -- 172.21.15.161:0/4226015459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64bc009b30 con 0x7f64c40d2a70 2024-03-23T14:37:43.700 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.692+0000 7f64c3fff640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c40c9770 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f64b000d5f0 tx=0x7f64b000dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.692+0000 7f64c17fa640 1 -- 172.21.15.161:0/4226015459 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b00125f0 con 0x7f64c40d2a70 2024-03-23T14:37:43.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.692+0000 7f64c17fa640 1 -- 172.21.15.161:0/4226015459 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64b0012750 con 0x7f64c40d2a70 2024-03-23T14:37:43.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.692+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64c40b7300 con 0x7f64c40d2a70 2024-03-23T14:37:43.701 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.692+0000 7f64c17fa640 1 -- 172.21.15.161:0/4226015459 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b0012b00 con 0x7f64c40d2a70 2024-03-23T14:37:43.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.692+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64c40b78c0 con 0x7f64c40d2a70 2024-03-23T14:37:43.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.693+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64c40b7a80 con 0x7f64c40d2a70 2024-03-23T14:37:43.702 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.693+0000 7f64c17fa640 1 -- 172.21.15.161:0/4226015459 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f64b0012c60 con 0x7f64c40d2a70 2024-03-23T14:37:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.694+0000 7f64c17fa640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f64a806b840 0x7f64a806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.694+0000 7f64c17fa640 1 -- 172.21.15.161:0/4226015459 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f64b008a370 con 0x7f64c40d2a70 2024-03-23T14:37:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.694+0000 7f64c8ada640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f64a806b840 0x7f64a806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.703 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.695+0000 7f64c8ada640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f64a806b840 0x7f64a806dd00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f64b8000f60 tx=0x7f64b8007b50 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.704 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.698+0000 7f64c17fa640 1 -- 172.21.15.161:0/4226015459 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f64b0055090 con 0x7f64c40d2a70 2024-03-23T14:37:43.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.873+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7f64c40cfa70 con 0x7f64a806b840 2024-03-23T14:37:43.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.874+0000 7f64c17fa640 1 -- 172.21.15.161:0/4226015459 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f64b8007170 con 0x7f64a806b840 2024-03-23T14:37:43.870 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 2 to deep-scrub 2024-03-23T14:37:43.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f64a806b840 msgr2=0x7f64a806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f64a806b840 0x7f64a806dd00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f64b8000f60 tx=0x7f64b8007b50 comp rx=0 tx=0).stop 2024-03-23T14:37:43.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 msgr2=0x7f64c40c9770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c40c9770 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f64b000d5f0 tx=0x7f64b000dab0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 shutdown_connections 2024-03-23T14:37:43.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f64a806b840 0x7f64a806dd00 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f64c4162970 0x7f64c40b2030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f64c40d2a70 0x7f64c40c9770 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 --2- 172.21.15.161:0/4226015459 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f64c40d2690 0x7f64c4152560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 >> 172.21.15.161:0/4226015459 conn(0x7f64c409d1b0 msgr2=0x7f64c40b3410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 shutdown_connections 2024-03-23T14:37:43.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.876+0000 7f64cad3e640 1 -- 172.21.15.161:0/4226015459 wait complete. 2024-03-23T14:37:43.883 INFO:tasks.ceph:Scrubbing osd.3 2024-03-23T14:37:43.883 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-03-23T14:37:43.987 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.992+0000 7fe404393640 1 Processor -- start 2024-03-23T14:37:43.988 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.992+0000 7fe404393640 1 -- start start 2024-03-23T14:37:43.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.992+0000 7fe404393640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc1446e0 0x7fe3fc06de40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.992+0000 7fe404393640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3fc06e380 0x7fe3fc071860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.992+0000 7fe404393640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fe3fc071da0 0x7fe3fc0b3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.992+0000 7fe404393640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fe3fc144ac0 con 0x7fe3fc06e380 2024-03-23T14:37:43.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.992+0000 7fe404393640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fe3fc15b750 con 0x7fe3fc071da0 2024-03-23T14:37:43.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.992+0000 7fe404393640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc072230 con 0x7fe3fc1446e0 2024-03-23T14:37:43.991 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40192e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3fc06e380 0x7fe3fc071860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40192e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fe3fc06e380 0x7fe3fc071860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38164/0 (socket says 172.21.15.161:38164) 2024-03-23T14:37:43.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40192e640 1 -- 172.21.15.161:0/547578663 learned_addr learned my addr 172.21.15.161:0/547578663 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:43.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe402930640 1 --2- 172.21.15.161:0/547578663 >> v2:172.21.15.183:3300/0 conn(0x7fe3fc071da0 0x7fe3fc0b3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40212f640 1 --2- 172.21.15.161:0/547578663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc1446e0 0x7fe3fc06de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:43.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40192e640 1 -- 172.21.15.161:0/547578663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc1446e0 msgr2=0x7fe3fc06de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40192e640 1 --2- 172.21.15.161:0/547578663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc1446e0 0x7fe3fc06de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40192e640 1 -- 172.21.15.161:0/547578663 >> v2:172.21.15.183:3300/0 conn(0x7fe3fc071da0 msgr2=0x7fe3fc0b3d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:43.992 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40192e640 1 --2- 172.21.15.161:0/547578663 >> v2:172.21.15.183:3300/0 conn(0x7fe3fc071da0 0x7fe3fc0b3d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.993+0000 7fe40192e640 1 -- 172.21.15.161:0/547578663 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3fc06e820 con 0x7fe3fc06e380 2024-03-23T14:37:43.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe40192e640 1 --2- 172.21.15.161:0/547578663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc06e380 0x7fe3fc071860 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0b4e70 tx=0x7fe3f802b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f2e6f2ea6534d5a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:43.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe40112d640 1 -- 172.21.15.161:0/547578663 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8031870 con 0x7fe3fc06e380 2024-03-23T14:37:43.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe40112d640 1 -- 172.21.15.161:0/547578663 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f80319d0 con 0x7fe3fc06e380 2024-03-23T14:37:43.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe40112d640 1 -- 172.21.15.161:0/547578663 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f8031d60 con 0x7fe3fc06e380 2024-03-23T14:37:43.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe404393640 1 -- 172.21.15.161:0/547578663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc06e380 msgr2=0x7fe3fc071860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:43.993 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe404393640 1 --2- 172.21.15.161:0/547578663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc06e380 0x7fe3fc071860 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fe3fc0b4e70 tx=0x7fe3f802b910 comp rx=0 tx=0).stop 2024-03-23T14:37:43.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe404393640 1 -- 172.21.15.161:0/547578663 shutdown_connections 2024-03-23T14:37:43.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe404393640 1 --2- 172.21.15.161:0/547578663 >> v2:172.21.15.183:3300/0 conn(0x7fe3fc071da0 0x7fe3fc0b3d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe404393640 1 --2- 172.21.15.161:0/547578663 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc06e380 0x7fe3fc071860 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe404393640 1 --2- 172.21.15.161:0/547578663 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc1446e0 0x7fe3fc06de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:43.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.994+0000 7fe404393640 1 -- 172.21.15.161:0/547578663 >> 172.21.15.161:0/547578663 conn(0x7fe3fc06b080 msgr2=0x7fe3fc06b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:43.994 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.995+0000 7fe404393640 1 -- 172.21.15.161:0/547578663 shutdown_connections 2024-03-23T14:37:43.995 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.995+0000 7fe404393640 1 -- 172.21.15.161:0/547578663 wait complete. 2024-03-23T14:37:43.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.996+0000 7fe404393640 1 Processor -- start 2024-03-23T14:37:43.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.996+0000 7fe404393640 1 -- start start 2024-03-23T14:37:43.998 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.996+0000 7fe404393640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc06e380 0x7fe3fc1e5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.996+0000 7fe404393640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3fc071da0 0x7fe3fc13d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.996+0000 7fe404393640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc1446e0 0x7fe3fc13de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:43.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.996+0000 7fe404393640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1e4ae0 con 0x7fe3fc1446e0 2024-03-23T14:37:43.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.996+0000 7fe404393640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fe3fc1e4cf0 con 0x7fe3fc071da0 2024-03-23T14:37:43.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.996+0000 7fe404393640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fe3fc1e4fd0 con 0x7fe3fc06e380 2024-03-23T14:37:43.999 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc1446e0 0x7fe3fc13de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc1446e0 0x7fe3fc13de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38166/0 (socket says 172.21.15.161:38166) 2024-03-23T14:37:44.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 -- 172.21.15.161:0/3279609760 learned_addr learned my addr 172.21.15.161:0/3279609760 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:44.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe40192e640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3fc071da0 0x7fe3fc13d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe40212f640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc06e380 0x7fe3fc1e5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 -- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc06e380 msgr2=0x7fe3fc1e5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fe3fc06e380 0x7fe3fc1e5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.000 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 -- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3fc071da0 msgr2=0x7fe3fc13d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3fc071da0 0x7fe3fc13d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 -- 172.21.15.161:0/3279609760 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3f80094d0 con 0x7fe3fc1446e0 2024-03-23T14:37:44.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe40192e640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fe3fc071da0 0x7fe3fc13d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:44.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe402930640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc1446e0 0x7fe3fc13de40 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4007db0 tx=0x7fe3f40106c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4013670 con 0x7fe3fc1446e0 2024-03-23T14:37:44.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe3f40137d0 con 0x7fe3fc1446e0 2024-03-23T14:37:44.001 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3fc1e52b0 con 0x7fe3fc1446e0 2024-03-23T14:37:44.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3f4013b80 con 0x7fe3fc1446e0 2024-03-23T14:37:44.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.997+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3fc1e5870 con 0x7fe3fc1446e0 2024-03-23T14:37:44.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.998+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe3fc15b3e0 con 0x7fe3fc1446e0 2024-03-23T14:37:44.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.999+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fe3f4013ce0 con 0x7fe3fc1446e0 2024-03-23T14:37:44.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.999+0000 7fe3eeffd640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe3e006b810 0x7fe3e006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.999+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7fe3f408cea0 con 0x7fe3fc1446e0 2024-03-23T14:37:44.002 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.999+0000 7fe3eeffd640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fe3e00716d0 0x7fe3e0073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.999+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 --> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe3e00741a0 con 0x7fe3e00716d0 2024-03-23T14:37:44.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.999+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fe3f408f560 con 0x7fe3fc1446e0 2024-03-23T14:37:44.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.999+0000 7fe40212f640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe3e006b810 0x7fe3e006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:43.999+0000 7fe40192e640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fe3e00716d0 0x7fe3e0073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.000+0000 7fe40192e640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fe3e00716d0 0x7fe3e0073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.000+0000 7fe40212f640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe3e006b810 0x7fe3e006dcd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0006fd0 tx=0x7fe3f0008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.004 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.000+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 <== osd.3 v2:172.21.15.161:6800/162554479 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe3e00741a0 con 0x7fe3e00716d0 2024-03-23T14:37:44.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.013+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 --> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe3fc06ede0 con 0x7fe3e00716d0 2024-03-23T14:37:44.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe3eeffd640 1 -- 172.21.15.161:0/3279609760 <== osd.3 v2:172.21.15.161:6800/162554479 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fe3f803a420 con 0x7fe3e00716d0 2024-03-23T14:37:44.010 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:44.010 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T14:37:44.010 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:44.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fe3e00716d0 msgr2=0x7fe3e0073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.161:6800/162554479,v1:172.21.15.161:6801/162554479] conn(0x7fe3e00716d0 0x7fe3e0073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe3e006b810 msgr2=0x7fe3e006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fe3e006b810 0x7fe3e006dcd0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7fe3f0006fd0 tx=0x7fe3f0008040 comp rx=0 tx=0).stop 2024-03-23T14:37:44.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc1446e0 msgr2=0x7fe3fc13de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 --2- 172.21.15.161:0/3279609760 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fe3fc1446e0 0x7fe3fc13de40 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fe3f4007db0 tx=0x7fe3f40106c0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe402930640 1 -- 172.21.15.161:0/3279609760 reap_dead start 2024-03-23T14:37:44.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 shutdown_connections 2024-03-23T14:37:44.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 >> 172.21.15.161:0/3279609760 conn(0x7fe3fc06b080 msgr2=0x7fe3fc0cbce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 shutdown_connections 2024-03-23T14:37:44.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.014+0000 7fe404393640 1 -- 172.21.15.161:0/3279609760 wait complete. 2024-03-23T14:37:44.020 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-03-23T14:37:44.123 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.127+0000 7f5af0e4e640 1 Processor -- start 2024-03-23T14:37:44.124 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.128+0000 7f5af0e4e640 1 -- start start 2024-03-23T14:37:44.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.128+0000 7f5af0e4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec0724e0 0x7f5aec06e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.128+0000 7f5af0e4e640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aec15be70 con 0x7f5aec072100 2024-03-23T14:37:44.127 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.128+0000 7f5af0e4e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aec15c150 con 0x7f5aec06ead0 2024-03-23T14:37:44.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.128+0000 7f5af0e4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec06ef80 con 0x7f5aec0724e0 2024-03-23T14:37:44.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.128+0000 7f5aea59c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f5aec072100 0x7f5aec15b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41068/0 (socket says 172.21.15.161:41068) 2024-03-23T14:37:44.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.128+0000 7f5aea59c640 1 -- 172.21.15.161:0/3090749850 learned_addr learned my addr 172.21.15.161:0/3090749850 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:44.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2270730863 0 0) 0x7f5aec15be70 con 0x7f5aec072100 2024-03-23T14:37:44.128 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae3fff640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec0724e0 0x7f5aec06e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ad8003520 con 0x7f5aec072100 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3164188584 0 0) 0x7f5aec15c150 con 0x7f5aec06ead0 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aec15be70 con 0x7f5aec06ead0 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3735088068 0 0) 0x7f5ad8003520 con 0x7f5aec072100 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec0724e0 msgr2=0x7f5aec06e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec0724e0 0x7f5aec06e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 >> v1:172.21.15.183:6789/0 conn(0x7f5aec06ead0 legacy=0x7f5aec0b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec0b2c70 con 0x7f5aec072100 2024-03-23T14:37:44.129 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f5adc001670 con 0x7f5aec072100 2024-03-23T14:37:44.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 >> v1:172.21.15.161:6789/0 conn(0x7f5aec072100 legacy=0x7f5aec15b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec0724e0 0x7f5ad8003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ad80040e0 0x7f5ad80047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ad8004d00 0x7f5ad8005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad800ba90 con 0x7f5ad80040e0 2024-03-23T14:37:44.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5ad800bc00 con 0x7f5ad8004d00 2024-03-23T14:37:44.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5ad800bdc0 con 0x7f5aec0724e0 2024-03-23T14:37:44.130 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae3fff640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ad80040e0 0x7f5ad80047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5aea59c640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec0724e0 0x7f5ad8003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5aead9d640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ad8004d00 0x7f5ad8005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae3fff640 1 -- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec0724e0 msgr2=0x7f5ad8003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:44.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae3fff640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec0724e0 0x7f5ad8003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae3fff640 1 -- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ad8004d00 msgr2=0x7f5ad8005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:44.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae3fff640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5ad8004d00 0x7f5ad8005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.129+0000 7f5ae3fff640 1 -- 172.21.15.161:0/3090749850 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ad8003520 con 0x7f5ad80040e0 2024-03-23T14:37:44.131 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.130+0000 7f5ae3fff640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ad80040e0 0x7f5ad80047c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009ac0 tx=0x7f5ad80057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc136f570f9b2661 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.130+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad402c600 con 0x7f5ad80040e0 2024-03-23T14:37:44.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.130+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ad402c760 con 0x7f5ad80040e0 2024-03-23T14:37:44.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.130+0000 7f5ae9d9b640 1 -- 172.21.15.161:0/3090749850 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ad402caf0 con 0x7f5ad80040e0 2024-03-23T14:37:44.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.131+0000 7f5af0e4e640 1 -- 172.21.15.161:0/3090749850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ad80040e0 msgr2=0x7f5ad80047c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.131+0000 7f5af0e4e640 1 --2- 172.21.15.161:0/3090749850 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5ad80040e0 0x7f5ad80047c0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7f5ad4009ac0 tx=0x7f5ad80057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.131+0000 7f5aead9d640 1 -- 172.21.15.161:0/3090749850 reap_dead start 2024-03-23T14:37:44.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.131+0000 7f5af0e4e640 1 -- 172.21.15.161:0/3090749850 shutdown_connections 2024-03-23T14:37:44.132 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.131+0000 7f5af0e4e640 1 -- 172.21.15.161:0/3090749850 >> 172.21.15.161:0/3090749850 conn(0x7f5aec0b17f0 msgr2=0x7f5aec0b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.131+0000 7f5af0e4e640 1 -- 172.21.15.161:0/3090749850 shutdown_connections 2024-03-23T14:37:44.133 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.131+0000 7f5af0e4e640 1 -- 172.21.15.161:0/3090749850 wait complete. 2024-03-23T14:37:44.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.132+0000 7f5af0e4e640 1 Processor -- start 2024-03-23T14:37:44.136 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.132+0000 7f5af0e4e640 1 -- start start 2024-03-23T14:37:44.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5af0e4e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aec1498e0 0x7f5aec1e5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5af0e4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec1e61e0 0x7f5aec159ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5af0e4e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec149cc0 0x7f5aec15a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5af0e4e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec1e7270 con 0x7f5aec149cc0 2024-03-23T14:37:44.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5af0e4e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f5aec1e6660 con 0x7f5aec1498e0 2024-03-23T14:37:44.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5af0e4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f5aec1e6940 con 0x7f5aec1e61e0 2024-03-23T14:37:44.137 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aead9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec149cc0 0x7f5aec15a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aead9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec149cc0 0x7f5aec15a400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38188/0 (socket says 172.21.15.161:38188) 2024-03-23T14:37:44.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aead9d640 1 -- 172.21.15.161:0/2834790294 learned_addr learned my addr 172.21.15.161:0/2834790294 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:44.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aea59c640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aec1498e0 0x7f5aec1e5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5ae3fff640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec1e61e0 0x7f5aec159ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aead9d640 1 -- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec1e61e0 msgr2=0x7f5aec159ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aead9d640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec1e61e0 0x7f5aec159ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aead9d640 1 -- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aec1498e0 msgr2=0x7f5aec1e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.138 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aead9d640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aec1498e0 0x7f5aec1e5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aead9d640 1 -- 172.21.15.161:0/2834790294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aec1e6c20 con 0x7f5aec149cc0 2024-03-23T14:37:44.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.133+0000 7f5aea59c640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aec1498e0 0x7f5aec1e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:44.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.134+0000 7f5aead9d640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec149cc0 0x7f5aec15a400 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f5aec070b40 tx=0x7f5ae400eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.134+0000 7f5ae2ffd640 1 -- 172.21.15.161:0/2834790294 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4006b50 con 0x7f5aec149cc0 2024-03-23T14:37:44.139 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.134+0000 7f5ae2ffd640 1 -- 172.21.15.161:0/2834790294 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ae4006cb0 con 0x7f5aec149cc0 2024-03-23T14:37:44.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.134+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5aec1e6f00 con 0x7f5aec149cc0 2024-03-23T14:37:44.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.134+0000 7f5ae2ffd640 1 -- 172.21.15.161:0/2834790294 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4003bb0 con 0x7f5aec149cc0 2024-03-23T14:37:44.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.134+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5aec0a97d0 con 0x7f5aec149cc0 2024-03-23T14:37:44.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.135+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5aec1e73e0 con 0x7f5aec149cc0 2024-03-23T14:37:44.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.135+0000 7f5ae2ffd640 1 -- 172.21.15.161:0/2834790294 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f5ae4003d10 con 0x7f5aec149cc0 2024-03-23T14:37:44.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.136+0000 7f5ae2ffd640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5acc06b810 0x7f5acc06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.140 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.136+0000 7f5ae2ffd640 1 -- 172.21.15.161:0/2834790294 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f5ae4089680 con 0x7f5aec149cc0 2024-03-23T14:37:44.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.136+0000 7f5aea59c640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5acc06b810 0x7f5acc06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.136+0000 7f5aea59c640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5acc06b810 0x7f5acc06dcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5adc001eb0 tx=0x7f5adc009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.141 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.140+0000 7f5ae2ffd640 1 -- 172.21.15.161:0/2834790294 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f5ae40543a0 con 0x7f5aec149cc0 2024-03-23T14:37:44.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.312+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7f5aec0735e0 con 0x7f5acc06b810 2024-03-23T14:37:44.309 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.313+0000 7f5ae2ffd640 1 -- 172.21.15.161:0/2834790294 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f5aec0735e0 con 0x7f5acc06b810 2024-03-23T14:37:44.309 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 3 to deep-scrub 2024-03-23T14:37:44.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5acc06b810 msgr2=0x7f5acc06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5acc06b810 0x7f5acc06dcd0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5adc001eb0 tx=0x7f5adc009040 comp rx=0 tx=0).stop 2024-03-23T14:37:44.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec149cc0 msgr2=0x7f5aec15a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec149cc0 0x7f5aec15a400 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7f5aec070b40 tx=0x7f5ae400eb20 comp rx=0 tx=0).stop 2024-03-23T14:37:44.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 shutdown_connections 2024-03-23T14:37:44.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f5acc06b810 0x7f5acc06dcd0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f5aec149cc0 0x7f5aec15a400 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f5aec1e61e0 0x7f5aec159ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 --2- 172.21.15.161:0/2834790294 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f5aec1498e0 0x7f5aec1e5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 >> 172.21.15.161:0/2834790294 conn(0x7f5aec0b17f0 msgr2=0x7f5aec0b09d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 shutdown_connections 2024-03-23T14:37:44.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.315+0000 7f5af0e4e640 1 -- 172.21.15.161:0/2834790294 wait complete. 2024-03-23T14:37:44.322 INFO:tasks.ceph:Scrubbing osd.4 2024-03-23T14:37:44.322 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-03-23T14:37:44.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.431+0000 7f4279984640 1 Processor -- start 2024-03-23T14:37:44.427 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.431+0000 7f4279984640 1 -- start start 2024-03-23T14:37:44.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f4279984640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f427415a740 0x7f427406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f4279984640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f427406ebd0 0x7f42740b3500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f4279984640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f42740cc740 con 0x7f427415a360 2024-03-23T14:37:44.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f4279984640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f42740cf710 con 0x7f427415a740 2024-03-23T14:37:44.431 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f4279984640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4274071740 con 0x7f427406ebd0 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f4272ffd640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f427415a360 0x7f427406d5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41082/0 (socket says 172.21.15.161:41082) 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f4272ffd640 1 -- 172.21.15.161:0/1141843988 learned_addr learned my addr 172.21.15.161:0/1141843988 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f426a7fc640 1 --2- 172.21.15.161:0/1141843988 >> v2:172.21.15.183:3300/0 conn(0x7f427415a740 0x7f427406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1925929102 0 0) 0x7f42740cc740 con 0x7f427415a360 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4260003440 con 0x7f427415a360 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.432+0000 7f42737fe640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f427406ebd0 0x7f42740b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 161830184 0 0) 0x7f4260003440 con 0x7f427415a360 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f427406ebd0 msgr2=0x7f42740b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:44.432 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f427406ebd0 0x7f42740b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 >> v2:172.21.15.183:3300/0 conn(0x7f427415a740 msgr2=0x7f427406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:44.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 --2- 172.21.15.161:0/1141843988 >> v2:172.21.15.183:3300/0 conn(0x7f427415a740 0x7f427406dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4274071a20 con 0x7f427415a360 2024-03-23T14:37:44.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f4264001690 con 0x7f427415a360 2024-03-23T14:37:44.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 >> v1:172.21.15.161:6789/0 conn(0x7f427415a360 legacy=0x7f427406d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4260003a60 0x7f4260003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f427406ebd0 0x7f426000a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.433 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426000adf0 0x7f426000d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f42740cf710 con 0x7f426000adf0 2024-03-23T14:37:44.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4274071740 con 0x7f4260003a60 2024-03-23T14:37:44.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f426000db80 con 0x7f427406ebd0 2024-03-23T14:37:44.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42737fe640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426000adf0 0x7f426000d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f4272ffd640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4260003a60 0x7f4260003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f426a7fc640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f427406ebd0 0x7f426000a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42737fe640 1 -- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f427406ebd0 msgr2=0x7f426000a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:44.434 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42737fe640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f427406ebd0 0x7f426000a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42737fe640 1 -- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4260003a60 msgr2=0x7f4260003e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:44.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42737fe640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4260003a60 0x7f4260003e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.433+0000 7f42737fe640 1 -- 172.21.15.161:0/1141843988 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4260003440 con 0x7f426000adf0 2024-03-23T14:37:44.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.434+0000 7f42737fe640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426000adf0 0x7f426000d280 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f426c009f90 tx=0x7f42600036f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a363449306cc43a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.434+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c02cbd0 con 0x7f426000adf0 2024-03-23T14:37:44.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.434+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426c00f040 con 0x7f426000adf0 2024-03-23T14:37:44.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.434+0000 7f42727fc640 1 -- 172.21.15.161:0/1141843988 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c034440 con 0x7f426000adf0 2024-03-23T14:37:44.435 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.435+0000 7f4279984640 1 -- 172.21.15.161:0/1141843988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426000adf0 msgr2=0x7f426000d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.435+0000 7f4279984640 1 --2- 172.21.15.161:0/1141843988 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f426000adf0 0x7f426000d280 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f426c009f90 tx=0x7f42600036f0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.435+0000 7f42737fe640 1 -- 172.21.15.161:0/1141843988 reap_dead start 2024-03-23T14:37:44.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.435+0000 7f4279984640 1 -- 172.21.15.161:0/1141843988 shutdown_connections 2024-03-23T14:37:44.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.435+0000 7f4279984640 1 -- 172.21.15.161:0/1141843988 >> 172.21.15.161:0/1141843988 conn(0x7f427406b080 msgr2=0x7f427406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.435+0000 7f4279984640 1 -- 172.21.15.161:0/1141843988 shutdown_connections 2024-03-23T14:37:44.436 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.435+0000 7f4279984640 1 -- 172.21.15.161:0/1141843988 wait complete. 2024-03-23T14:37:44.439 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.436+0000 7f4279984640 1 Processor -- start 2024-03-23T14:37:44.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.436+0000 7f4279984640 1 -- start start 2024-03-23T14:37:44.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4279984640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4274148090 0x7f4274158200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4279984640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4274157a10 0x7f427415c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4279984640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42741589f0 0x7f427415c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4279984640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f4274156c90 con 0x7f42741589f0 2024-03-23T14:37:44.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4279984640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f4274148470 con 0x7f4274148090 2024-03-23T14:37:44.440 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4279984640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f4274158f90 con 0x7f4274157a10 2024-03-23T14:37:44.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42741589f0 0x7f427415c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42741589f0 0x7f427415c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38198/0 (socket says 172.21.15.161:38198) 2024-03-23T14:37:44.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 -- 172.21.15.161:0/4161804005 learned_addr learned my addr 172.21.15.161:0/4161804005 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:44.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4272ffd640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4274148090 0x7f4274158200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f426a7fc640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4274157a10 0x7f427415c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 -- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4274157a10 msgr2=0x7f427415c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.441 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f4274157a10 0x7f427415c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 -- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4274148090 msgr2=0x7f4274158200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f4274148090 0x7f4274158200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 -- 172.21.15.161:0/4161804005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4274156f30 con 0x7f42741589f0 2024-03-23T14:37:44.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f42737fe640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42741589f0 0x7f427415c7b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f426c037040 tx=0x7f426c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c043070 con 0x7f42741589f0 2024-03-23T14:37:44.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f426c02de60 con 0x7f42741589f0 2024-03-23T14:37:44.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426c034d90 con 0x7f42741589f0 2024-03-23T14:37:44.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.437+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4274157210 con 0x7f42741589f0 2024-03-23T14:37:44.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.438+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4274158740 con 0x7f42741589f0 2024-03-23T14:37:44.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.438+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4274071d00 con 0x7f42741589f0 2024-03-23T14:37:44.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.439+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f426c03e440 con 0x7f42741589f0 2024-03-23T14:37:44.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.439+0000 7f4270ff9640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f425406b810 0x7f425406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.439+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f426c0ab920 con 0x7f42741589f0 2024-03-23T14:37:44.443 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.439+0000 7f4270ff9640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7f42540716d0 0x7f4254073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.439+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 --> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f42540741a0 con 0x7f42540716d0 2024-03-23T14:37:44.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.439+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f426c0b02c0 con 0x7f42741589f0 2024-03-23T14:37:44.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.439+0000 7f4272ffd640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f425406b810 0x7f425406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.440+0000 7f426a7fc640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7f42540716d0 0x7f4254073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.440+0000 7f4272ffd640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f425406b810 0x7f425406dcd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4264000c00 tx=0x7f42640078f0 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.440+0000 7f426a7fc640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7f42540716d0 0x7f4254073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.444 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.441+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 <== osd.4 v2:172.21.15.183:6820/3154566470 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f42540741a0 con 0x7f42540716d0 2024-03-23T14:37:44.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.453+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 --> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f4274070fa0 con 0x7f42540716d0 2024-03-23T14:37:44.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.454+0000 7f4270ff9640 1 -- 172.21.15.161:0/4161804005 <== osd.4 v2:172.21.15.183:6820/3154566470 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f425c0098b0 con 0x7f42540716d0 2024-03-23T14:37:44.450 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:44.450 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T14:37:44.451 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:44.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7f42540716d0 msgr2=0x7f4254073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6820/3154566470,v1:172.21.15.183:6822/3154566470] conn(0x7f42540716d0 0x7f4254073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f425406b810 msgr2=0x7f425406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f425406b810 0x7f425406dcd0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4264000c00 tx=0x7f42640078f0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42741589f0 msgr2=0x7f427415c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 --2- 172.21.15.161:0/4161804005 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f42741589f0 0x7f427415c7b0 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f426c037040 tx=0x7f426c009750 comp rx=0 tx=0).stop 2024-03-23T14:37:44.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f42737fe640 1 -- 172.21.15.161:0/4161804005 reap_dead start 2024-03-23T14:37:44.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 shutdown_connections 2024-03-23T14:37:44.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 >> 172.21.15.161:0/4161804005 conn(0x7f427406b080 msgr2=0x7f42741487d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 shutdown_connections 2024-03-23T14:37:44.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.455+0000 7f4279984640 1 -- 172.21.15.161:0/4161804005 wait complete. 2024-03-23T14:37:44.461 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-03-23T14:37:44.564 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.566+0000 7f922fad4640 1 Processor -- start 2024-03-23T14:37:44.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.566+0000 7f922fad4640 1 -- start start 2024-03-23T14:37:44.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922fad4640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f922815e400 0x7f92280cf490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922fad4640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92280d2b40 0x7f92280b0a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.565 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922fad4640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92280d0460 con 0x7f92280cf9d0 2024-03-23T14:37:44.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922fad4640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f92280d1050 con 0x7f92280d2b40 2024-03-23T14:37:44.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922fad4640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f92280d06b0 con 0x7f922815e400 2024-03-23T14:37:44.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922e071640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92280d2b40 0x7f92280b0a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922e071640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f92280d2b40 0x7f92280b0a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38212/0 (socket says 172.21.15.161:38212) 2024-03-23T14:37:44.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922d06f640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f92280cf9d0 0x7f92280d2430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41096/0 (socket says 172.21.15.161:41096) 2024-03-23T14:37:44.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.567+0000 7f922e071640 1 -- 172.21.15.161:0/955749334 learned_addr learned my addr 172.21.15.161:0/955749334 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:44.566 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.568+0000 7f922d870640 1 --2- 172.21.15.161:0/955749334 >> v2:172.21.15.183:3300/0 conn(0x7f922815e400 0x7f92280cf490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.568+0000 7f922e071640 1 -- 172.21.15.161:0/955749334 >> v2:172.21.15.183:3300/0 conn(0x7f922815e400 msgr2=0x7f92280cf490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:44.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.568+0000 7f922e071640 1 --2- 172.21.15.161:0/955749334 >> v2:172.21.15.183:3300/0 conn(0x7f922815e400 0x7f92280cf490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.568+0000 7f922e071640 1 -- 172.21.15.161:0/955749334 >> v1:172.21.15.161:6789/0 conn(0x7f92280cf9d0 legacy=0x7f92280d2430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.568+0000 7f922e071640 1 -- 172.21.15.161:0/955749334 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92280d0990 con 0x7f92280d2b40 2024-03-23T14:37:44.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.568+0000 7f922c86e640 1 -- 172.21.15.161:0/955749334 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217264117 0 0) 0x7f92280d0460 con 0x7f92280cf9d0 2024-03-23T14:37:44.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.568+0000 7f922e071640 1 --2- 172.21.15.161:0/955749334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280b0a80 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f922808e230 tx=0x7f922002b9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d34fff6384d32681 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922c86e640 1 -- 172.21.15.161:0/955749334 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9220030860 con 0x7f92280d2b40 2024-03-23T14:37:44.567 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922c86e640 1 -- 172.21.15.161:0/955749334 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f92200309c0 con 0x7f92280d2b40 2024-03-23T14:37:44.568 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922c86e640 1 -- 172.21.15.161:0/955749334 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9220030d50 con 0x7f92280d2b40 2024-03-23T14:37:44.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922fad4640 1 -- 172.21.15.161:0/955749334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 msgr2=0x7f92280b0a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922fad4640 1 --2- 172.21.15.161:0/955749334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280b0a80 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f922808e230 tx=0x7f922002b9a0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922fad4640 1 -- 172.21.15.161:0/955749334 shutdown_connections 2024-03-23T14:37:44.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922fad4640 1 --2- 172.21.15.161:0/955749334 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280b0a80 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922fad4640 1 --2- 172.21.15.161:0/955749334 >> v2:172.21.15.183:3300/0 conn(0x7f922815e400 0x7f92280cf490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.571 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922fad4640 1 -- 172.21.15.161:0/955749334 >> 172.21.15.161:0/955749334 conn(0x7f922809d190 msgr2=0x7f92280d3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922fad4640 1 -- 172.21.15.161:0/955749334 shutdown_connections 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.569+0000 7f922fad4640 1 -- 172.21.15.161:0/955749334 wait complete. 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.570+0000 7f922fad4640 1 Processor -- start 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.570+0000 7f922fad4640 1 -- start start 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922fad4640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92280cf9d0 0x7f92281593b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922fad4640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280caf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922fad4640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f922815e400 0x7f922815b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922fad4640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f92280b72b0 con 0x7f92280d2b40 2024-03-23T14:37:44.572 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922fad4640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f92281598f0 con 0x7f92280cf9d0 2024-03-23T14:37:44.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922fad4640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f92280cb9e0 con 0x7f922815e400 2024-03-23T14:37:44.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922d06f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280caf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922d06f640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280caf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38224/0 (socket says 172.21.15.161:38224) 2024-03-23T14:37:44.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922d06f640 1 -- 172.21.15.161:0/259155308 learned_addr learned my addr 172.21.15.161:0/259155308 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:44.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922d06f640 1 -- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f922815e400 msgr2=0x7f922815b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922d06f640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f922815e400 0x7f922815b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922d06f640 1 -- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92280cf9d0 msgr2=0x7f92281593b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.573 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922d06f640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92280cf9d0 0x7f92281593b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.571+0000 7f922d06f640 1 -- 172.21.15.161:0/259155308 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92200094d0 con 0x7f92280d2b40 2024-03-23T14:37:44.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.572+0000 7f922d06f640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280caf20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f921800c4d0 tx=0x7f921800c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.572+0000 7f921e7fc640 1 -- 172.21.15.161:0/259155308 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92180074e0 con 0x7f92280d2b40 2024-03-23T14:37:44.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.572+0000 7f921e7fc640 1 -- 172.21.15.161:0/259155308 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9218007660 con 0x7f92280d2b40 2024-03-23T14:37:44.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.572+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92280cbcc0 con 0x7f92280d2b40 2024-03-23T14:37:44.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.572+0000 7f921e7fc640 1 -- 172.21.15.161:0/259155308 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9218004a20 con 0x7f92280d2b40 2024-03-23T14:37:44.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.572+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92281608b0 con 0x7f92280d2b40 2024-03-23T14:37:44.574 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.573+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9228160e40 con 0x7f92280d2b40 2024-03-23T14:37:44.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.573+0000 7f921e7fc640 1 -- 172.21.15.161:0/259155308 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9218004b80 con 0x7f92280d2b40 2024-03-23T14:37:44.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.573+0000 7f921e7fc640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f920c06b810 0x7f920c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.574+0000 7f921e7fc640 1 -- 172.21.15.161:0/259155308 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f92180893b0 con 0x7f92280d2b40 2024-03-23T14:37:44.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.574+0000 7f922d870640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f920c06b810 0x7f920c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.574+0000 7f922d870640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f920c06b810 0x7f920c06dcd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f92240070b0 tx=0x7f9224007040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.575 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.578+0000 7f921e7fc640 1 -- 172.21.15.161:0/259155308 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9218054160 con 0x7f92280d2b40 2024-03-23T14:37:44.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.753+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f92280cc1e0 con 0x7f920c06b810 2024-03-23T14:37:44.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.753+0000 7f921e7fc640 1 -- 172.21.15.161:0/259155308 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f92280cc1e0 con 0x7f920c06b810 2024-03-23T14:37:44.749 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 4 to deep-scrub 2024-03-23T14:37:44.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f920c06b810 msgr2=0x7f920c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f920c06b810 0x7f920c06dcd0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f92240070b0 tx=0x7f9224007040 comp rx=0 tx=0).stop 2024-03-23T14:37:44.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 msgr2=0x7f92280caf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280caf20 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f921800c4d0 tx=0x7f921800c990 comp rx=0 tx=0).stop 2024-03-23T14:37:44.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 shutdown_connections 2024-03-23T14:37:44.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f920c06b810 0x7f920c06dcd0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f922815e400 0x7f922815b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f92280d2b40 0x7f92280caf20 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 --2- 172.21.15.161:0/259155308 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f92280cf9d0 0x7f92281593b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 >> 172.21.15.161:0/259155308 conn(0x7f922809d190 msgr2=0x7f92280b0f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 shutdown_connections 2024-03-23T14:37:44.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.756+0000 7f922fad4640 1 -- 172.21.15.161:0/259155308 wait complete. 2024-03-23T14:37:44.762 INFO:tasks.ceph:Scrubbing osd.5 2024-03-23T14:37:44.762 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-03-23T14:37:44.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.871+0000 7fa0ac29d640 1 Processor -- start 2024-03-23T14:37:44.867 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.871+0000 7fa0ac29d640 1 -- start start 2024-03-23T14:37:44.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0ac29d640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa0a4144aa0 0x7fa0a4070b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0ac29d640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a4144e80 con 0x7fa0a40d0160 2024-03-23T14:37:44.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0ac29d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa0a40b3800 con 0x7fa0a40710d0 2024-03-23T14:37:44.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0ac29d640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa0a40b3ae0 con 0x7fa0a4144aa0 2024-03-23T14:37:44.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0aa83a640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fa0a40d0160 0x7fa0a40ceca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41104/0 (socket says 172.21.15.161:41104) 2024-03-23T14:37:44.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0aa83a640 1 -- 172.21.15.161:0/3552985600 learned_addr learned my addr 172.21.15.161:0/3552985600 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:44.869 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0aa039640 1 --2- 172.21.15.161:0/3552985600 >> v2:172.21.15.183:3300/0 conn(0x7fa0a4144aa0 0x7fa0a4070b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0a9037640 1 -- 172.21.15.161:0/3552985600 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2885716015 0 0) 0x7fa0a4144e80 con 0x7fa0a40d0160 2024-03-23T14:37:44.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.872+0000 7fa0a9037640 1 -- 172.21.15.161:0/3552985600 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa090003520 con 0x7fa0a40d0160 2024-03-23T14:37:44.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.873+0000 7fa0aa039640 1 -- 172.21.15.161:0/3552985600 >> v1:172.21.15.183:6789/0 conn(0x7fa0a40710d0 legacy=0x7fa0a40cfa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.873+0000 7fa0a9037640 1 -- 172.21.15.161:0/3552985600 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3745249854 0 0) 0x7fa0a40b3800 con 0x7fa0a40710d0 2024-03-23T14:37:44.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.873+0000 7fa0aa039640 1 -- 172.21.15.161:0/3552985600 >> v1:172.21.15.161:6789/0 conn(0x7fa0a40d0160 legacy=0x7fa0a40ceca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.870 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.873+0000 7fa0aa039640 1 -- 172.21.15.161:0/3552985600 --> v2:172.21.15.183:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a40ce4a0 con 0x7fa0a4144aa0 2024-03-23T14:37:44.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.873+0000 7fa0aa039640 1 --2- 172.21.15.161:0/3552985600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 0x7fa0a4070b90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009d90 tx=0x7fa0a002be20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=437afd0c710b9548 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0a9037640 1 -- 172.21.15.161:0/3552985600 <== mon.1 v2:172.21.15.183:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a002cbd0 con 0x7fa0a4144aa0 2024-03-23T14:37:44.873 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0a9037640 1 -- 172.21.15.161:0/3552985600 <== mon.1 v2:172.21.15.183:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa0a002cd30 con 0x7fa0a4144aa0 2024-03-23T14:37:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0a9037640 1 -- 172.21.15.161:0/3552985600 <== mon.1 v2:172.21.15.183:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0a002cea0 con 0x7fa0a4144aa0 2024-03-23T14:37:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3552985600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 msgr2=0x7fa0a4070b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0ac29d640 1 --2- 172.21.15.161:0/3552985600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 0x7fa0a4070b90 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0009d90 tx=0x7fa0a002be20 comp rx=0 tx=0).stop 2024-03-23T14:37:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3552985600 shutdown_connections 2024-03-23T14:37:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0ac29d640 1 --2- 172.21.15.161:0/3552985600 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 0x7fa0a4070b90 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3552985600 >> 172.21.15.161:0/3552985600 conn(0x7fa0a406b080 msgr2=0x7fa0a406b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3552985600 shutdown_connections 2024-03-23T14:37:44.874 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.874+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3552985600 wait complete. 2024-03-23T14:37:44.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.875+0000 7fa0ac29d640 1 Processor -- start 2024-03-23T14:37:44.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0ac29d640 1 -- start start 2024-03-23T14:37:44.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0ac29d640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0a40d0160 0x7fa0a406cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0ac29d640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 0x7fa0a415b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0ac29d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0a41f7460 0x7fa0a40c6630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0ac29d640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a406d130 con 0x7fa0a41f7460 2024-03-23T14:37:44.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0ac29d640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa0a406d3d0 con 0x7fa0a4144aa0 2024-03-23T14:37:44.875 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0ac29d640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa0a406d6b0 con 0x7fa0a40d0160 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0aa83a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0a41f7460 0x7fa0a40c6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0aa83a640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0a41f7460 0x7fa0a40c6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38230/0 (socket says 172.21.15.161:38230) 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0aa83a640 1 -- 172.21.15.161:0/3143570189 learned_addr learned my addr 172.21.15.161:0/3143570189 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.876+0000 7fa0a9838640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 0x7fa0a415b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0aa039640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0a40d0160 0x7fa0a406cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0aa83a640 1 -- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0a40d0160 msgr2=0x7fa0a406cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0aa83a640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa0a40d0160 0x7fa0a406cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0aa83a640 1 -- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 msgr2=0x7fa0a415b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.876 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0aa83a640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 0x7fa0a415b6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0aa83a640 1 -- 172.21.15.161:0/3143570189 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa0a0009a70 con 0x7fa0a41f7460 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0a9838640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa0a4144aa0 0x7fa0a415b6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0aa83a640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0a41f7460 0x7fa0a40c6630 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00c400 tx=0x7fa09c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09c005ce0 con 0x7fa0a41f7460 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa09c005e40 con 0x7fa0a41f7460 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa09c0072d0 con 0x7fa0a41f7460 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa0a406d990 con 0x7fa0a41f7460 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.877+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa0a406df50 con 0x7fa0a41f7460 2024-03-23T14:37:44.877 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.878+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa0a40cf3b0 con 0x7fa0a41f7460 2024-03-23T14:37:44.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.878+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa09c0074f0 con 0x7fa0a41f7460 2024-03-23T14:37:44.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.879+0000 7fa09affd640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa08806b810 0x7fa08806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.879 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.879+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7fa09c089360 con 0x7fa0a41f7460 2024-03-23T14:37:44.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.879+0000 7fa09affd640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa0880716d0 0x7fa088073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:44.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.879+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 --> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa0880741a0 con 0x7fa0880716d0 2024-03-23T14:37:44.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.879+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fa09c08ba20 con 0x7fa0a41f7460 2024-03-23T14:37:44.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.879+0000 7fa0aa039640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa08806b810 0x7fa08806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.879+0000 7fa0a9838640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa0880716d0 0x7fa088073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:44.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.880+0000 7fa0a9838640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa0880716d0 0x7fa088073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.880+0000 7fa0aa039640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa08806b810 0x7fa08806dcd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0036000 tx=0x7fa0a0030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:44.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.880+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 <== osd.5 v2:172.21.15.183:6816/2304513884 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa0880741a0 con 0x7fa0880716d0 2024-03-23T14:37:44.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.893+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 --> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa0a40c4a60 con 0x7fa0880716d0 2024-03-23T14:37:44.890 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa09affd640 1 -- 172.21.15.161:0/3143570189 <== osd.5 v2:172.21.15.183:6816/2304513884 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa0a40c4a60 con 0x7fa0880716d0 2024-03-23T14:37:44.890 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:44.890 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T14:37:44.890 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:44.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa0880716d0 msgr2=0x7fa088073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.891 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6816/2304513884,v1:172.21.15.183:6817/2304513884] conn(0x7fa0880716d0 0x7fa088073af0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa08806b810 msgr2=0x7fa08806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa08806b810 0x7fa08806dcd0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fa0a0036000 tx=0x7fa0a0030000 comp rx=0 tx=0).stop 2024-03-23T14:37:44.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0a41f7460 msgr2=0x7fa0a40c6630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:44.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 --2- 172.21.15.161:0/3143570189 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa0a41f7460 0x7fa0a40c6630 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fa09c00c400 tx=0x7fa09c00c8c0 comp rx=0 tx=0).stop 2024-03-23T14:37:44.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0aa83a640 1 -- 172.21.15.161:0/3143570189 reap_dead start 2024-03-23T14:37:44.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 shutdown_connections 2024-03-23T14:37:44.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 >> 172.21.15.161:0/3143570189 conn(0x7fa0a406b080 msgr2=0x7fa0a413f620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:44.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.894+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 shutdown_connections 2024-03-23T14:37:44.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:44.895+0000 7fa0ac29d640 1 -- 172.21.15.161:0/3143570189 wait complete. 2024-03-23T14:37:44.901 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-03-23T14:37:45.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.007+0000 7f9ab636c640 1 Processor -- start 2024-03-23T14:37:45.003 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.007+0000 7f9ab636c640 1 -- start start 2024-03-23T14:37:45.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9ab636c640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9ab00724e0 0x7f9ab006e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9ab636c640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f9ab006ead0 0x7f9ab00b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9ab636c640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ab015be70 con 0x7f9ab0072100 2024-03-23T14:37:45.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9ab636c640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab006ef80 con 0x7f9ab00724e0 2024-03-23T14:37:45.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9ab636c640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f9ab015c430 con 0x7f9ab006ead0 2024-03-23T14:37:45.006 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaf7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9ab00724e0 0x7f9ab006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaffff640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f9ab0072100 0x7f9ab015b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41116/0 (socket says 172.21.15.161:41116) 2024-03-23T14:37:45.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaf7fe640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f9ab00724e0 0x7f9ab006e590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38244/0 (socket says 172.21.15.161:38244) 2024-03-23T14:37:45.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaffff640 1 -- 172.21.15.161:0/4232782272 learned_addr learned my addr 172.21.15.161:0/4232782272 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:45.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaf7fe640 1 -- 172.21.15.161:0/4232782272 >> v2:172.21.15.183:3300/0 conn(0x7f9ab006ead0 msgr2=0x7f9ab00b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9ab4909640 1 --2- 172.21.15.161:0/4232782272 >> v2:172.21.15.183:3300/0 conn(0x7f9ab006ead0 0x7f9ab00b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaf7fe640 1 --2- 172.21.15.161:0/4232782272 >> v2:172.21.15.183:3300/0 conn(0x7f9ab006ead0 0x7f9ab00b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaf7fe640 1 -- 172.21.15.161:0/4232782272 >> v1:172.21.15.161:6789/0 conn(0x7f9ab0072100 legacy=0x7f9ab015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.007 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaeffd640 1 -- 172.21.15.161:0/4232782272 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 142364050 0 0) 0x7f9ab015be70 con 0x7f9ab0072100 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.008+0000 7f9aaf7fe640 1 -- 172.21.15.161:0/4232782272 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ab015c150 con 0x7f9ab00724e0 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.009+0000 7f9aaf7fe640 1 --2- 172.21.15.161:0/4232782272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab00724e0 0x7f9ab006e590 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9ab00b27b0 tx=0x7f9aa002e0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8328018b625ce541 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.009+0000 7f9aaeffd640 1 -- 172.21.15.161:0/4232782272 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa002ee50 con 0x7f9ab00724e0 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.009+0000 7f9aaeffd640 1 -- 172.21.15.161:0/4232782272 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa0036440 con 0x7f9ab00724e0 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.009+0000 7f9aaeffd640 1 -- 172.21.15.161:0/4232782272 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa00367d0 con 0x7f9ab00724e0 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.010+0000 7f9ab636c640 1 -- 172.21.15.161:0/4232782272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab00724e0 msgr2=0x7f9ab006e590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.010+0000 7f9ab636c640 1 --2- 172.21.15.161:0/4232782272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab00724e0 0x7f9ab006e590 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f9ab00b27b0 tx=0x7f9aa002e0a0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.010+0000 7f9ab636c640 1 -- 172.21.15.161:0/4232782272 shutdown_connections 2024-03-23T14:37:45.008 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.010+0000 7f9ab636c640 1 --2- 172.21.15.161:0/4232782272 >> v2:172.21.15.183:3300/0 conn(0x7f9ab006ead0 0x7f9ab00b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.010+0000 7f9ab636c640 1 --2- 172.21.15.161:0/4232782272 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab00724e0 0x7f9ab006e590 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.010+0000 7f9ab636c640 1 -- 172.21.15.161:0/4232782272 >> 172.21.15.161:0/4232782272 conn(0x7f9ab00b17f0 msgr2=0x7f9ab00b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.010+0000 7f9ab636c640 1 -- 172.21.15.161:0/4232782272 shutdown_connections 2024-03-23T14:37:45.009 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.010+0000 7f9ab636c640 1 -- 172.21.15.161:0/4232782272 wait complete. 2024-03-23T14:37:45.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.011+0000 7f9ab636c640 1 Processor -- start 2024-03-23T14:37:45.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.011+0000 7f9ab636c640 1 -- start start 2024-03-23T14:37:45.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab636c640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ab006ead0 0x7f9ab0147cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab636c640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab0072100 0x7f9ab00ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab636c640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ab00724e0 0x7f9ab00aeec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.012 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab636c640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab00af400 con 0x7f9ab0072100 2024-03-23T14:37:45.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab636c640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f9ab014aa90 con 0x7f9ab006ead0 2024-03-23T14:37:45.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab636c640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f9ab00aa3b0 con 0x7f9ab00724e0 2024-03-23T14:37:45.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab0072100 0x7f9ab00ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab0072100 0x7f9ab00ae980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38252/0 (socket says 172.21.15.161:38252) 2024-03-23T14:37:45.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 -- 172.21.15.161:0/769888215 learned_addr learned my addr 172.21.15.161:0/769888215 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:45.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaffff640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ab006ead0 0x7f9ab0147cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab4909640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ab00724e0 0x7f9ab00aeec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.013 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 -- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ab00724e0 msgr2=0x7f9ab00aeec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ab00724e0 0x7f9ab00aeec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 -- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ab006ead0 msgr2=0x7f9ab0147cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ab006ead0 0x7f9ab0147cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 -- 172.21.15.161:0/769888215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa000b6a0 con 0x7f9ab0072100 2024-03-23T14:37:45.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaffff640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ab006ead0 0x7f9ab0147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:45.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aaf7fe640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab0072100 0x7f9ab00ae980 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aacff9640 1 -- 172.21.15.161:0/769888215 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa003e7d0 con 0x7f9ab0072100 2024-03-23T14:37:45.014 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aacff9640 1 -- 172.21.15.161:0/769888215 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9aa003e930 con 0x7f9ab0072100 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ab00aa690 con 0x7f9ab0072100 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9aacff9640 1 -- 172.21.15.161:0/769888215 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9aa003eca0 con 0x7f9ab0072100 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.012+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ab00aac50 con 0x7f9ab0072100 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.013+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ab015baa0 con 0x7f9ab0072100 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.014+0000 7f9aacff9640 1 -- 172.21.15.161:0/769888215 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f9aa003d070 con 0x7f9ab0072100 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.014+0000 7f9aacff9640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9a9c06b810 0x7f9a9c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.015+0000 7f9aacff9640 1 -- 172.21.15.161:0/769888215 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f9aa00ac210 con 0x7f9ab0072100 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.015+0000 7f9aaffff640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9a9c06b810 0x7f9a9c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.015 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.015+0000 7f9aaffff640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9a9c06b810 0x7f9a9c06dcd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4000f60 tx=0x7f9aa4007c10 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.016 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.019+0000 7f9aacff9640 1 -- 172.21.15.161:0/769888215 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f9aa0076f30 con 0x7f9ab0072100 2024-03-23T14:37:45.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.191+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f9ab0070d00 con 0x7f9a9c06b810 2024-03-23T14:37:45.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.191+0000 7f9aacff9640 1 -- 172.21.15.161:0/769888215 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f9aa4007270 con 0x7f9a9c06b810 2024-03-23T14:37:45.187 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 5 to deep-scrub 2024-03-23T14:37:45.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9a9c06b810 msgr2=0x7f9a9c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9a9c06b810 0x7f9a9c06dcd0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4000f60 tx=0x7f9aa4007c10 comp rx=0 tx=0).stop 2024-03-23T14:37:45.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab0072100 msgr2=0x7f9ab00ae980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab0072100 0x7f9ab00ae980 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7f9aa0000c00 tx=0x7f9aa00046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 shutdown_connections 2024-03-23T14:37:45.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f9a9c06b810 0x7f9a9c06dcd0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f9ab00724e0 0x7f9ab00aeec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f9ab0072100 0x7f9ab00ae980 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 --2- 172.21.15.161:0/769888215 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f9ab006ead0 0x7f9ab0147cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 >> 172.21.15.161:0/769888215 conn(0x7f9ab00b17f0 msgr2=0x7f9ab00b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 shutdown_connections 2024-03-23T14:37:45.191 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.194+0000 7f9ab636c640 1 -- 172.21.15.161:0/769888215 wait complete. 2024-03-23T14:37:45.200 INFO:tasks.ceph:Scrubbing osd.6 2024-03-23T14:37:45.200 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-03-23T14:37:45.307 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.311+0000 7f2a1a0cd640 1 Processor -- start 2024-03-23T14:37:45.308 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.311+0000 7f2a1a0cd640 1 -- start start 2024-03-23T14:37:45.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a1a0cd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1415e550 0x7f2a140ceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a1a0cd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2a140cfca0 0x7f2a140ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a1a0cd640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f2a140cf530 0x7f2a140d05b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.310 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a1a0cd640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f2a140d1e10 con 0x7f2a140cfca0 2024-03-23T14:37:45.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a1a0cd640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f2a140cf9a0 con 0x7f2a140cf530 2024-03-23T14:37:45.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a1a0cd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2a140d4970 con 0x7f2a1415e550 2024-03-23T14:37:45.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a0affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2a140cfca0 0x7f2a140ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a0affd640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f2a140cfca0 0x7f2a140ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38264/0 (socket says 172.21.15.161:38264) 2024-03-23T14:37:45.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.312+0000 7f2a0affd640 1 -- 172.21.15.161:0/3618818751 learned_addr learned my addr 172.21.15.161:0/3618818751 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:45.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.313+0000 7f2a13fff640 1 --2- 172.21.15.161:0/3618818751 >> v2:172.21.15.183:3300/0 conn(0x7f2a140cf530 0x7f2a140d05b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.313+0000 7f2a0affd640 1 -- 172.21.15.161:0/3618818751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1415e550 msgr2=0x7f2a140ceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.311 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.313+0000 7f2a0affd640 1 --2- 172.21.15.161:0/3618818751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1415e550 0x7f2a140ceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.313+0000 7f2a0affd640 1 -- 172.21.15.161:0/3618818751 >> v2:172.21.15.183:3300/0 conn(0x7f2a140cf530 msgr2=0x7f2a140d05b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.313+0000 7f2a0affd640 1 --2- 172.21.15.161:0/3618818751 >> v2:172.21.15.183:3300/0 conn(0x7f2a140cf530 0x7f2a140d05b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.313+0000 7f2a0affd640 1 -- 172.21.15.161:0/3618818751 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a140d4c50 con 0x7f2a140cfca0 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.313+0000 7f2a13fff640 1 --2- 172.21.15.161:0/3618818751 >> v2:172.21.15.183:3300/0 conn(0x7f2a140cf530 0x7f2a140d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.313+0000 7f2a0affd640 1 --2- 172.21.15.161:0/3618818751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cfca0 0x7f2a140ceff0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2a140d19f0 tx=0x7f29fc02b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d93d66393734007 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a12ffd640 1 -- 172.21.15.161:0/3618818751 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc0317e0 con 0x7f2a140cfca0 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a12ffd640 1 -- 172.21.15.161:0/3618818751 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29fc031940 con 0x7f2a140cfca0 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a12ffd640 1 -- 172.21.15.161:0/3618818751 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29fc031cd0 con 0x7f2a140cfca0 2024-03-23T14:37:45.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/3618818751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cfca0 msgr2=0x7f2a140ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 --2- 172.21.15.161:0/3618818751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cfca0 0x7f2a140ceff0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f2a140d19f0 tx=0x7f29fc02b880 comp rx=0 tx=0).stop 2024-03-23T14:37:45.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/3618818751 shutdown_connections 2024-03-23T14:37:45.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 --2- 172.21.15.161:0/3618818751 >> v2:172.21.15.183:3300/0 conn(0x7f2a140cf530 0x7f2a140d05b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 --2- 172.21.15.161:0/3618818751 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cfca0 0x7f2a140ceff0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 --2- 172.21.15.161:0/3618818751 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a1415e550 0x7f2a140ceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/3618818751 >> 172.21.15.161:0/3618818751 conn(0x7f2a1409d210 msgr2=0x7f2a140d3f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/3618818751 shutdown_connections 2024-03-23T14:37:45.314 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.314+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/3618818751 wait complete. 2024-03-23T14:37:45.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.315+0000 7f2a1a0cd640 1 Processor -- start 2024-03-23T14:37:45.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.315+0000 7f2a1a0cd640 1 -- start start 2024-03-23T14:37:45.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a1a0cd640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cf530 0x7f2a14153410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a1a0cd640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a140cfca0 0x7f2a140cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a1a0cd640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a1415e550 0x7f2a140cd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a1a0cd640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f2a14153950 con 0x7f2a140cf530 2024-03-23T14:37:45.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a1a0cd640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f2a140b7bd0 con 0x7f2a1415e550 2024-03-23T14:37:45.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a1a0cd640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f2a140b6b30 con 0x7f2a140cfca0 2024-03-23T14:37:45.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cf530 0x7f2a14153410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a137fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cf530 0x7f2a14153410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38280/0 (socket says 172.21.15.161:38280) 2024-03-23T14:37:45.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a137fe640 1 -- 172.21.15.161:0/4158078170 learned_addr learned my addr 172.21.15.161:0/4158078170 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:45.318 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a13fff640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a1415e550 0x7f2a140cd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.316+0000 7f2a0affd640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a140cfca0 0x7f2a140cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a137fe640 1 -- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a140cfca0 msgr2=0x7f2a140cd2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a137fe640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a140cfca0 0x7f2a140cd2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a137fe640 1 -- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a1415e550 msgr2=0x7f2a140cd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a137fe640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a1415e550 0x7f2a140cd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a137fe640 1 -- 172.21.15.161:0/4158078170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29fc0094d0 con 0x7f2a140cf530 2024-03-23T14:37:45.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a13fff640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f2a1415e550 0x7f2a140cd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:45.319 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a0affd640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f2a140cfca0 0x7f2a140cd2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:37:45.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a137fe640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cf530 0x7f2a14153410 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400b410 tx=0x7f2a0400b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a04011830 con 0x7f2a140cf530 2024-03-23T14:37:45.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a040119b0 con 0x7f2a140cf530 2024-03-23T14:37:45.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a140b6e10 con 0x7f2a140cf530 2024-03-23T14:37:45.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a04011d60 con 0x7f2a140cf530 2024-03-23T14:37:45.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.317+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a140b73d0 con 0x7f2a140cf530 2024-03-23T14:37:45.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.318+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2a140d4620 con 0x7f2a140cf530 2024-03-23T14:37:45.320 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.318+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f2a0401a400 con 0x7f2a140cf530 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.319+0000 7f2a10ff9640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f29f406b810 0x7f29f406dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.319+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f2a0408a6e0 con 0x7f2a140cf530 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.319+0000 7f2a10ff9640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7f29f40716d0 0x7f29f4073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.319+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 --> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f29f40741a0 con 0x7f29f40716d0 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.319+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7f2a0408ab30 con 0x7f2a140cf530 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.319+0000 7f2a0affd640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f29f406b810 0x7f29f406dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.319+0000 7f2a13fff640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7f29f40716d0 0x7f29f4073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.320+0000 7f2a0affd640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f29f406b810 0x7f29f406dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f29fc036000 tx=0x7f29fc03d040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.321 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.320+0000 7f2a13fff640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7f29f40716d0 0x7f29f4073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.322 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.320+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 <== osd.6 v2:172.21.15.183:6800/963839040 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f29f40741a0 con 0x7f29f40716d0 2024-03-23T14:37:45.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.333+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 --> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2a140b7840 con 0x7f29f40716d0 2024-03-23T14:37:45.329 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a10ff9640 1 -- 172.21.15.161:0/4158078170 <== osd.6 v2:172.21.15.183:6800/963839040 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2a140b7840 con 0x7f29f40716d0 2024-03-23T14:37:45.330 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:45.330 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T14:37:45.330 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:45.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7f29f40716d0 msgr2=0x7f29f4073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6800/963839040,v1:172.21.15.183:6801/963839040] conn(0x7f29f40716d0 0x7f29f4073af0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f29f406b810 msgr2=0x7f29f406dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.331 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f29f406b810 0x7f29f406dcd0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f29fc036000 tx=0x7f29fc03d040 comp rx=0 tx=0).stop 2024-03-23T14:37:45.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cf530 msgr2=0x7f2a14153410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 --2- 172.21.15.161:0/4158078170 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f2a140cf530 0x7f2a14153410 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7f2a0400b410 tx=0x7f2a0400b8d0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a13fff640 1 -- 172.21.15.161:0/4158078170 reap_dead start 2024-03-23T14:37:45.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 shutdown_connections 2024-03-23T14:37:45.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 >> 172.21.15.161:0/4158078170 conn(0x7f2a1409d210 msgr2=0x7f2a140cea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 shutdown_connections 2024-03-23T14:37:45.332 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.334+0000 7f2a1a0cd640 1 -- 172.21.15.161:0/4158078170 wait complete. 2024-03-23T14:37:45.340 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-03-23T14:37:45.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.446+0000 7efdad7a2640 1 Processor -- start 2024-03-23T14:37:45.442 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.446+0000 7efdad7a2640 1 -- start start 2024-03-23T14:37:45.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efdad7a2640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7efda815e220 0x7efda806eae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efdad7a2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 0x7efda806f020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.449 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efdad7a2640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efda815e600 con 0x7efda80ad1b0 2024-03-23T14:37:45.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efdad7a2640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7efda806e330 con 0x7efda815e220 2024-03-23T14:37:45.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efdad7a2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efda806e610 con 0x7efda806dcd0 2024-03-23T14:37:45.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda77fe640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7efda80ad1b0 0x7efda806d5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41120/0 (socket says 172.21.15.161:41120) 2024-03-23T14:37:45.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda77fe640 1 -- 172.21.15.161:0/1718416986 learned_addr learned my addr 172.21.15.161:0/1718416986 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:45.450 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda7fff640 1 --2- 172.21.15.161:0/1718416986 >> v2:172.21.15.183:3300/0 conn(0x7efda815e220 0x7efda806eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efdacfa1640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 0x7efda806f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4207099715 0 0) 0x7efd9c001110 con 0x7efda80ad1b0 2024-03-23T14:37:45.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd90003520 con 0x7efda80ad1b0 2024-03-23T14:37:45.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3984874856 0 0) 0x7efd90003520 con 0x7efda80ad1b0 2024-03-23T14:37:45.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 msgr2=0x7efda806f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 0x7efda806f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.451 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 >> v2:172.21.15.183:3300/0 conn(0x7efda815e220 msgr2=0x7efda806eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 --2- 172.21.15.161:0/1718416986 >> v2:172.21.15.183:3300/0 conn(0x7efda815e220 0x7efda806eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda80cf760 con 0x7efda80ad1b0 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.447+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7efd9c001800 con 0x7efda80ad1b0 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 >> v1:172.21.15.161:6789/0 conn(0x7efda80ad1b0 legacy=0x7efda806d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda6ffd640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd90003970 0x7efd90003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda6ffd640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 0x7efd900047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda6ffd640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd90004d00 0x7efd90005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efda806e330 con 0x7efd90003970 2024-03-23T14:37:45.452 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efda806e610 con 0x7efd90004d00 2024-03-23T14:37:45.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efd9000b980 con 0x7efda806dcd0 2024-03-23T14:37:45.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda7fff640 1 --2- 172.21.15.161:0/1718416986 >> v2:172.21.15.183:3300/0 conn(0x7efda815e220 0x7efda806eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-23T14:37:45.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda7fff640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd90003970 0x7efd90003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda77fe640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 0x7efd900047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efdacfa1640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd90004d00 0x7efd90005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda7fff640 1 -- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 msgr2=0x7efd900047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda7fff640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 0x7efd900047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.453 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda7fff640 1 -- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd90004d00 msgr2=0x7efd90005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda7fff640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd90004d00 0x7efd90005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda7fff640 1 -- 172.21.15.161:0/1718416986 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd90003520 con 0x7efd90003970 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efdacfa1640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efd90004d00 0x7efd90005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.448+0000 7efda77fe640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efda806dcd0 0x7efd900047c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efda7fff640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd90003970 0x7efd90003d90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7efd98009c70 tx=0x7efd900057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8d0a568d10f818 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd98004020 con 0x7efd90003970 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd98004180 con 0x7efd90003970 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efda6ffd640 1 -- 172.21.15.161:0/1718416986 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9802c960 con 0x7efd90003970 2024-03-23T14:37:45.454 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efdad7a2640 1 -- 172.21.15.161:0/1718416986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd90003970 msgr2=0x7efd90003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efdad7a2640 1 --2- 172.21.15.161:0/1718416986 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efd90003970 0x7efd90003d90 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7efd98009c70 tx=0x7efd900057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efdacfa1640 1 -- 172.21.15.161:0/1718416986 reap_dead start 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efdad7a2640 1 -- 172.21.15.161:0/1718416986 shutdown_connections 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efdad7a2640 1 -- 172.21.15.161:0/1718416986 >> 172.21.15.161:0/1718416986 conn(0x7efda8071980 msgr2=0x7efda80b1ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efdad7a2640 1 -- 172.21.15.161:0/1718416986 shutdown_connections 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.449+0000 7efdad7a2640 1 -- 172.21.15.161:0/1718416986 wait complete. 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.451+0000 7efdad7a2640 1 Processor -- start 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.451+0000 7efdad7a2640 1 -- start start 2024-03-23T14:37:45.455 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.451+0000 7efdad7a2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd90003970 0x7efda8099970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.451+0000 7efdad7a2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efda8159b70 0x7efda81601d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.451+0000 7efdad7a2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efda80cfed0 0x7efda80d0350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.451+0000 7efdad7a2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efda80d0890 con 0x7efda8159b70 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.451+0000 7efdad7a2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efda8099eb0 con 0x7efda80cfed0 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.451+0000 7efdad7a2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efda80b30d0 con 0x7efd90003970 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efda8159b70 0x7efda81601d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efda8159b70 0x7efda81601d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38304/0 (socket says 172.21.15.161:38304) 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 -- 172.21.15.161:0/2276954065 learned_addr learned my addr 172.21.15.161:0/2276954065 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:45.456 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda7fff640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd90003970 0x7efda8099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efdacfa1640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efda80cfed0 0x7efda80d0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 -- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd90003970 msgr2=0x7efda8099970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd90003970 0x7efda8099970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 -- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efda80cfed0 msgr2=0x7efda80d0350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efda80cfed0 0x7efda80d0350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 -- 172.21.15.161:0/2276954065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efda80b33b0 con 0x7efda8159b70 2024-03-23T14:37:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda77fe640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efda8159b70 0x7efda81601d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7efd9c000fb0 tx=0x7efd9c007790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.457 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda4ff9640 1 -- 172.21.15.161:0/2276954065 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00e710 con 0x7efda8159b70 2024-03-23T14:37:45.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda4ff9640 1 -- 172.21.15.161:0/2276954065 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd9c00e870 con 0x7efda8159b70 2024-03-23T14:37:45.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efda80b3690 con 0x7efda8159b70 2024-03-23T14:37:45.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efda4ff9640 1 -- 172.21.15.161:0/2276954065 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd9c00ec00 con 0x7efda8159b70 2024-03-23T14:37:45.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.452+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efda80b3c50 con 0x7efda8159b70 2024-03-23T14:37:45.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.453+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efda80b3e40 con 0x7efda8159b70 2024-03-23T14:37:45.458 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.454+0000 7efda4ff9640 1 -- 172.21.15.161:0/2276954065 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efd9c01d460 con 0x7efda8159b70 2024-03-23T14:37:45.459 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.454+0000 7efda4ff9640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd9406b840 0x7efd9406dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.454+0000 7efda4ff9640 1 -- 172.21.15.161:0/2276954065 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7efd9c089c80 con 0x7efda8159b70 2024-03-23T14:37:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.454+0000 7efda7fff640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd9406b840 0x7efd9406dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.455+0000 7efda7fff640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd9406b840 0x7efd9406dd00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7efd98035000 tx=0x7efd98030000 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.460 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.458+0000 7efda4ff9640 1 -- 172.21.15.161:0/2276954065 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efd9c054a50 con 0x7efda8159b70 2024-03-23T14:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.631+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7efda80cf410 con 0x7efd9406b840 2024-03-23T14:37:45.627 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.632+0000 7efda4ff9640 1 -- 172.21.15.161:0/2276954065 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7efd980043f0 con 0x7efd9406b840 2024-03-23T14:37:45.628 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 6 to deep-scrub 2024-03-23T14:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd9406b840 msgr2=0x7efd9406dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd9406b840 0x7efd9406dd00 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7efd98035000 tx=0x7efd98030000 comp rx=0 tx=0).stop 2024-03-23T14:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efda8159b70 msgr2=0x7efda81601d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.630 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efda8159b70 0x7efda81601d0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7efd9c000fb0 tx=0x7efd9c007790 comp rx=0 tx=0).stop 2024-03-23T14:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 shutdown_connections 2024-03-23T14:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efd9406b840 0x7efd9406dd00 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efda80cfed0 0x7efda80d0350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efda8159b70 0x7efda81601d0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 --2- 172.21.15.161:0/2276954065 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efd90003970 0x7efda8099970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 >> 172.21.15.161:0/2276954065 conn(0x7efda8071980 msgr2=0x7efda80ae810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 shutdown_connections 2024-03-23T14:37:45.631 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.634+0000 7efdad7a2640 1 -- 172.21.15.161:0/2276954065 wait complete. 2024-03-23T14:37:45.640 INFO:tasks.ceph:Scrubbing osd.7 2024-03-23T14:37:45.641 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-03-23T14:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.750+0000 7fa42ec4e640 1 Processor -- start 2024-03-23T14:37:45.746 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.750+0000 7fa42ec4e640 1 -- start start 2024-03-23T14:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42ec4e640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fa4280cfa50 0x7fa4280cfeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42ec4e640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa4280ceca0 0x7fa4280cf130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.747 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42ec4e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4280d03f0 con 0x7fa428144aa0 2024-03-23T14:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42ec4e640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fa4280cf670 con 0x7fa4280ceca0 2024-03-23T14:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42ec4e640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fa428144e80 con 0x7fa4280cfa50 2024-03-23T14:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42d1eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa4280ceca0 0x7fa4280cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42d1eb640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fa4280ceca0 0x7fa4280cf130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38318/0 (socket says 172.21.15.161:38318) 2024-03-23T14:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42d1eb640 1 -- 172.21.15.161:0/4231264555 learned_addr learned my addr 172.21.15.161:0/4231264555 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa427fff640 1 --2- 172.21.15.161:0/4231264555 >> v2:172.21.15.183:3300/0 conn(0x7fa4280cfa50 0x7fa4280cfeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42d1eb640 1 -- 172.21.15.161:0/4231264555 >> v2:172.21.15.183:3300/0 conn(0x7fa4280cfa50 msgr2=0x7fa4280cfeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.748 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42d1eb640 1 --2- 172.21.15.161:0/4231264555 >> v2:172.21.15.183:3300/0 conn(0x7fa4280cfa50 0x7fa4280cfeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42d1eb640 1 -- 172.21.15.161:0/4231264555 >> v1:172.21.15.183:6789/0 conn(0x7fa428144aa0 legacy=0x7fa428070b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.749 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.751+0000 7fa42d1eb640 1 -- 172.21.15.161:0/4231264555 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4280712a0 con 0x7fa4280ceca0 2024-03-23T14:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.752+0000 7fa42d1eb640 1 --2- 172.21.15.161:0/4231264555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280ceca0 0x7fa4280cf130 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa428072210 tx=0x7fa42002bdc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=426f8783c696c141 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.752+0000 7fa4277fe640 1 -- 172.21.15.161:0/4231264555 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42002c960 con 0x7fa4280ceca0 2024-03-23T14:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.752+0000 7fa4277fe640 1 -- 172.21.15.161:0/4231264555 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa42002cac0 con 0x7fa4280ceca0 2024-03-23T14:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.752+0000 7fa4277fe640 1 -- 172.21.15.161:0/4231264555 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa42002ce50 con 0x7fa4280ceca0 2024-03-23T14:37:45.752 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.753+0000 7fa42ec4e640 1 -- 172.21.15.161:0/4231264555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280ceca0 msgr2=0x7fa4280cf130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.753+0000 7fa42ec4e640 1 --2- 172.21.15.161:0/4231264555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280ceca0 0x7fa4280cf130 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7fa428072210 tx=0x7fa42002bdc0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.753+0000 7fa42ec4e640 1 -- 172.21.15.161:0/4231264555 shutdown_connections 2024-03-23T14:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.753+0000 7fa42ec4e640 1 --2- 172.21.15.161:0/4231264555 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280ceca0 0x7fa4280cf130 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.753+0000 7fa42ec4e640 1 --2- 172.21.15.161:0/4231264555 >> v2:172.21.15.183:3300/0 conn(0x7fa4280cfa50 0x7fa4280cfeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.753+0000 7fa42ec4e640 1 -- 172.21.15.161:0/4231264555 >> 172.21.15.161:0/4231264555 conn(0x7fa42806b080 msgr2=0x7fa42806b460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.753+0000 7fa42ec4e640 1 -- 172.21.15.161:0/4231264555 shutdown_connections 2024-03-23T14:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.753+0000 7fa42ec4e640 1 -- 172.21.15.161:0/4231264555 wait complete. 2024-03-23T14:37:45.753 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.754+0000 7fa42ec4e640 1 Processor -- start 2024-03-23T14:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.754+0000 7fa42ec4e640 1 -- start start 2024-03-23T14:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa42ec4e640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4280ceca0 0x7fa4281f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa42ec4e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280cfa50 0x7fa4280b1800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa42ec4e640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428144aa0 0x7fa4280b1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa42ec4e640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fa42813caf0 con 0x7fa4280cfa50 2024-03-23T14:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa42ec4e640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fa4281f4c10 con 0x7fa4280ceca0 2024-03-23T14:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa42ec4e640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fa42813dae0 con 0x7fa428144aa0 2024-03-23T14:37:45.754 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280cfa50 0x7fa4280b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280cfa50 0x7fa4280b1800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38330/0 (socket says 172.21.15.161:38330) 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 -- 172.21.15.161:0/2900772822 learned_addr learned my addr 172.21.15.161:0/2900772822 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa42c9ea640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4280ceca0 0x7fa4281f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa42d1eb640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428144aa0 0x7fa4280b1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 -- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428144aa0 msgr2=0x7fa4280b1d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fa428144aa0 0x7fa4280b1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 -- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4280ceca0 msgr2=0x7fa4281f3440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fa4280ceca0 0x7fa4281f3440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.755 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 -- 172.21.15.161:0/2900772822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa420009b30 con 0x7fa4280cfa50 2024-03-23T14:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa427fff640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280cfa50 0x7fa4280b1800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa41400d5d0 tx=0x7fa41400da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa4140125f0 con 0x7fa4280cfa50 2024-03-23T14:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.755+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa414012770 con 0x7fa4280cfa50 2024-03-23T14:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.756+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa414012b20 con 0x7fa4280cfa50 2024-03-23T14:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.756+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa42813cd20 con 0x7fa4280cfa50 2024-03-23T14:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.756+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa42813d2e0 con 0x7fa4280cfa50 2024-03-23T14:37:45.756 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.756+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa4280ce4a0 con 0x7fa4280cfa50 2024-03-23T14:37:45.757 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.757+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fa414012c80 con 0x7fa4280cfa50 2024-03-23T14:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.757+0000 7fa4257fa640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40c06b810 0x7fa40c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.757+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7fa41408a410 con 0x7fa4280cfa50 2024-03-23T14:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.758+0000 7fa42c9ea640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40c06b810 0x7fa40c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.758+0000 7fa4257fa640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7fa40c0716d0 0x7fa40c073af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.758+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 --> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa40c0741a0 con 0x7fa40c0716d0 2024-03-23T14:37:45.758 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.758+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_get_version_reply(handle=1 version=51) ==== 24+0+0 (secure 0 0 0) 0x7fa41408cad0 con 0x7fa4280cfa50 2024-03-23T14:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.758+0000 7fa42d1eb640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7fa40c0716d0 0x7fa40c073af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.758+0000 7fa42c9ea640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40c06b810 0x7fa40c06dcd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa41c007fd0 tx=0x7fa41c009040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.758+0000 7fa42d1eb640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7fa40c0716d0 0x7fa40c073af0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.759 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.759+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 <== osd.7 v2:172.21.15.183:6808/4120799670 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa40c0741a0 con 0x7fa40c0716d0 2024-03-23T14:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.772+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 --> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa4280b3ae0 con 0x7fa40c0716d0 2024-03-23T14:37:45.768 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa4257fa640 1 -- 172.21.15.161:0/2900772822 <== osd.7 v2:172.21.15.183:6808/4120799670 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa42002c950 con 0x7fa40c0716d0 2024-03-23T14:37:45.769 INFO:teuthology.orchestra.run.smithi161.stdout:{ 2024-03-23T14:37:45.769 INFO:teuthology.orchestra.run.smithi161.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-03-23T14:37:45.769 INFO:teuthology.orchestra.run.smithi161.stdout:} 2024-03-23T14:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7fa40c0716d0 msgr2=0x7fa40c073af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6808/4120799670,v1:172.21.15.183:6809/4120799670] conn(0x7fa40c0716d0 0x7fa40c073af0 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40c06b810 msgr2=0x7fa40c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fa40c06b810 0x7fa40c06dcd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fa41c007fd0 tx=0x7fa41c009040 comp rx=0 tx=0).stop 2024-03-23T14:37:45.770 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280cfa50 msgr2=0x7fa4280b1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 --2- 172.21.15.161:0/2900772822 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fa4280cfa50 0x7fa4280b1800 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7fa41400d5d0 tx=0x7fa41400da90 comp rx=0 tx=0).stop 2024-03-23T14:37:45.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42d1eb640 1 -- 172.21.15.161:0/2900772822 reap_dead start 2024-03-23T14:37:45.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 shutdown_connections 2024-03-23T14:37:45.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 >> 172.21.15.161:0/2900772822 conn(0x7fa42806b080 msgr2=0x7fa428070b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 shutdown_connections 2024-03-23T14:37:45.771 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.773+0000 7fa42ec4e640 1 -- 172.21.15.161:0/2900772822 wait complete. 2024-03-23T14:37:45.780 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-03-23T14:37:45.880 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.884+0000 7f7e75015640 1 Processor -- start 2024-03-23T14:37:45.881 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.884+0000 7f7e75015640 1 -- start start 2024-03-23T14:37:45.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.885+0000 7f7e75015640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e700724e0 0x7f7e7006e590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.885+0000 7f7e75015640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f7e7006ead0 0x7f7e700b3470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.885+0000 7f7e75015640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7e7015be70 con 0x7f7e70072100 2024-03-23T14:37:45.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.885+0000 7f7e75015640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f7e7006ef80 con 0x7f7e7006ead0 2024-03-23T14:37:45.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.885+0000 7f7e75015640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7e7015c430 con 0x7f7e700724e0 2024-03-23T14:37:45.884 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.885+0000 7f7e6ed9d640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f7e70072100 0x7f7e7015b3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41128/0 (socket says 172.21.15.161:41128) 2024-03-23T14:37:45.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.885+0000 7f7e6ed9d640 1 -- 172.21.15.161:0/1835698060 learned_addr learned my addr 172.21.15.161:0/1835698060 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:37:45.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2593336126 0 0) 0x7f7e7015be70 con 0x7f7e70072100 2024-03-23T14:37:45.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6f59e640 1 --2- 172.21.15.161:0/1835698060 >> v2:172.21.15.183:3300/0 conn(0x7f7e7006ead0 0x7f7e700b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7e58003520 con 0x7f7e70072100 2024-03-23T14:37:45.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6e59c640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e700724e0 0x7f7e7006e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3042715618 0 0) 0x7f7e58003520 con 0x7f7e70072100 2024-03-23T14:37:45.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e700724e0 msgr2=0x7f7e7006e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.885 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e700724e0 0x7f7e7006e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 >> v2:172.21.15.183:3300/0 conn(0x7f7e7006ead0 msgr2=0x7f7e700b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 --2- 172.21.15.161:0/1835698060 >> v2:172.21.15.183:3300/0 conn(0x7f7e7006ead0 0x7f7e700b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e7015c150 con 0x7f7e70072100 2024-03-23T14:37:45.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f7e680019f0 con 0x7f7e70072100 2024-03-23T14:37:45.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 >> v1:172.21.15.161:6789/0 conn(0x7f7e70072100 legacy=0x7f7e7015b3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e58003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e700724e0 0x7f7e580047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.886 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e58004d00 0x7f7e58005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7e7015c430 con 0x7f7e58003970 2024-03-23T14:37:45.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7e5800b980 con 0x7f7e58004d00 2024-03-23T14:37:45.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7e5800bac0 con 0x7f7e700724e0 2024-03-23T14:37:45.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6ed9d640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e58003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6e59c640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e700724e0 0x7f7e580047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6f59e640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e58004d00 0x7f7e58005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6ed9d640 1 -- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e700724e0 msgr2=0x7f7e580047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.887 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6ed9d640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e700724e0 0x7f7e580047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6ed9d640 1 -- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e58004d00 msgr2=0x7f7e58005180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:45.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6ed9d640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e58004d00 0x7f7e58005180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6ed9d640 1 -- 172.21.15.161:0/1835698060 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e58003520 con 0x7f7e58003970 2024-03-23T14:37:45.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.886+0000 7f7e6f59e640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e58004d00 0x7f7e58005180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:45.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.887+0000 7f7e6ed9d640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e58003d90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800a8d0 tx=0x7f7e580057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2185df63323731a server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.887+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68034630 con 0x7f7e58003970 2024-03-23T14:37:45.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.887+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e6802f070 con 0x7f7e58003970 2024-03-23T14:37:45.888 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.887+0000 7f7e6dd9b640 1 -- 172.21.15.161:0/1835698060 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680349d0 con 0x7f7e58003970 2024-03-23T14:37:45.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.888+0000 7f7e75015640 1 -- 172.21.15.161:0/1835698060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 msgr2=0x7f7e58003d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.888+0000 7f7e75015640 1 --2- 172.21.15.161:0/1835698060 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e58003d90 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800a8d0 tx=0x7f7e580057f0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.888+0000 7f7e6f59e640 1 -- 172.21.15.161:0/1835698060 reap_dead start 2024-03-23T14:37:45.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.888+0000 7f7e75015640 1 -- 172.21.15.161:0/1835698060 shutdown_connections 2024-03-23T14:37:45.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.888+0000 7f7e75015640 1 -- 172.21.15.161:0/1835698060 >> 172.21.15.161:0/1835698060 conn(0x7f7e700b17f0 msgr2=0x7f7e700b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:45.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.888+0000 7f7e75015640 1 -- 172.21.15.161:0/1835698060 shutdown_connections 2024-03-23T14:37:45.889 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.888+0000 7f7e75015640 1 -- 172.21.15.161:0/1835698060 wait complete. 2024-03-23T14:37:45.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.889+0000 7f7e75015640 1 Processor -- start 2024-03-23T14:37:45.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.889+0000 7f7e75015640 1 -- start start 2024-03-23T14:37:45.892 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e75015640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e70155e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e75015640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7014ca90 0x7f7e701536e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e75015640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e70154fd0 0x7f7e70139020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e75015640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f7e70154b40 con 0x7f7e58003970 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e75015640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f7e70153f30 con 0x7f7e70154fd0 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e75015640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f7e70154210 con 0x7f7e7014ca90 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e70155e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6ed9d640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e70155e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38360/0 (socket says 172.21.15.161:38360) 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6ed9d640 1 -- 172.21.15.161:0/3747396774 learned_addr learned my addr 172.21.15.161:0/3747396774 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:45.893 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6e59c640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7014ca90 0x7f7e701536e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6f59e640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e70154fd0 0x7f7e70139020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6ed9d640 1 -- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7014ca90 msgr2=0x7f7e701536e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6ed9d640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7014ca90 0x7f7e701536e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6ed9d640 1 -- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e70154fd0 msgr2=0x7f7e70139020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:45.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6ed9d640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e70154fd0 0x7f7e70139020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:45.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.890+0000 7f7e6ed9d640 1 -- 172.21.15.161:0/3747396774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e701544f0 con 0x7f7e58003970 2024-03-23T14:37:45.894 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.891+0000 7f7e6ed9d640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e70155e70 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800a9f0 tx=0x7f7e68001360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.891+0000 7f7e577fe640 1 -- 172.21.15.161:0/3747396774 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e680348c0 con 0x7f7e58003970 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.891+0000 7f7e577fe640 1 -- 172.21.15.161:0/3747396774 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7e68034a40 con 0x7f7e58003970 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.891+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e701547d0 con 0x7f7e58003970 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.891+0000 7f7e577fe640 1 -- 172.21.15.161:0/3747396774 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e68034df0 con 0x7f7e58003970 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.891+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e7014cf10 con 0x7f7e58003970 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.892+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7e700735e0 con 0x7f7e58003970 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.892+0000 7f7e577fe640 1 -- 172.21.15.161:0/3747396774 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f7e68031030 con 0x7f7e58003970 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.893+0000 7f7e577fe640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7e5c06b840 0x7f7e5c06dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:45.895 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.893+0000 7f7e577fe640 1 -- 172.21.15.161:0/3747396774 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f7e680abf70 con 0x7f7e58003970 2024-03-23T14:37:45.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.893+0000 7f7e6e59c640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7e5c06b840 0x7f7e5c06dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:45.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.893+0000 7f7e6e59c640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7e5c06b840 0x7f7e5c06dd00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7e700cec30 tx=0x7f7e64008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:45.896 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:45.897+0000 7f7e577fe640 1 -- 172.21.15.161:0/3747396774 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f7e6807a7c0 con 0x7f7e58003970 2024-03-23T14:37:46.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.068+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f7e700b3c00 con 0x7f7e5c06b840 2024-03-23T14:37:46.064 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.069+0000 7f7e577fe640 1 -- 172.21.15.161:0/3747396774 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f7e700b3c00 con 0x7f7e5c06b840 2024-03-23T14:37:46.064 INFO:teuthology.orchestra.run.smithi161.stderr:instructed osd(s) 7 to deep-scrub 2024-03-23T14:37:46.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7e5c06b840 msgr2=0x7f7e5c06dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:46.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7e5c06b840 0x7f7e5c06dd00 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f7e700cec30 tx=0x7f7e64008040 comp rx=0 tx=0).stop 2024-03-23T14:37:46.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 msgr2=0x7f7e70155e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:46.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e70155e70 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f7e6800a9f0 tx=0x7f7e68001360 comp rx=0 tx=0).stop 2024-03-23T14:37:46.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 shutdown_connections 2024-03-23T14:37:46.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f7e5c06b840 0x7f7e5c06dd00 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.067 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f7e70154fd0 0x7f7e70139020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f7e7014ca90 0x7f7e701536e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 --2- 172.21.15.161:0/3747396774 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f7e58003970 0x7f7e70155e70 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 >> 172.21.15.161:0/3747396774 conn(0x7f7e700b17f0 msgr2=0x7f7e700b0460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:46.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 shutdown_connections 2024-03-23T14:37:46.068 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.071+0000 7f7e75015640 1 -- 172.21.15.161:0/3747396774 wait complete. 2024-03-23T14:37:46.077 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:37:46.178 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.182+0000 7f68a03c7640 1 Processor -- start 2024-03-23T14:37:46.179 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.182+0000 7f68a03c7640 1 -- start start 2024-03-23T14:37:46.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.183+0000 7f68a03c7640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f68980b4260 0x7f689815b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:46.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.183+0000 7f68a03c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b3610 0x7f68980b3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:46.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.183+0000 7f68a03c7640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6898070b90 con 0x7f689806eb30 2024-03-23T14:37:46.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.183+0000 7f68a03c7640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7f68980b3fe0 con 0x7f68980b4260 2024-03-23T14:37:46.181 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.183+0000 7f68a03c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f689815ab90 con 0x7f68980b3610 2024-03-23T14:37:46.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689d962640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f68980b4260 0x7f689815b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:46.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689d962640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7f68980b4260 0x7f689815b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38364/0 (socket says 172.21.15.161:38364) 2024-03-23T14:37:46.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689e163640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f689806eb30 0x7f68980b4870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:41138/0 (socket says 172.21.15.161:41138) 2024-03-23T14:37:46.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689d962640 1 -- 172.21.15.161:0/1580114759 learned_addr learned my addr 172.21.15.161:0/1580114759 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:46.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689e964640 1 --2- 172.21.15.161:0/1580114759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b3610 0x7f68980b3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:46.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689d962640 1 -- 172.21.15.161:0/1580114759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b3610 msgr2=0x7f68980b3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:37:46.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689d962640 1 --2- 172.21.15.161:0/1580114759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b3610 0x7f68980b3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.182 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689d962640 1 -- 172.21.15.161:0/1580114759 >> v1:172.21.15.161:6789/0 conn(0x7f689806eb30 legacy=0x7f68980b4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689d962640 1 -- 172.21.15.161:0/1580114759 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f689815b0d0 con 0x7f68980b4260 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689d161640 1 -- 172.21.15.161:0/1580114759 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 987254522 0 0) 0x7f6898070b90 con 0x7f689806eb30 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.184+0000 7f689e964640 1 --2- 172.21.15.161:0/1580114759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b3610 0x7f68980b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f689d962640 1 --2- 172.21.15.161:0/1580114759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b4260 0x7f689815b560 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f688800bb10 tx=0x7f688802e150 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6787087e0662a0dc server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f689d161640 1 -- 172.21.15.161:0/1580114759 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6888036440 con 0x7f68980b4260 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f689d161640 1 -- 172.21.15.161:0/1580114759 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f68880365a0 con 0x7f68980b4260 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f689d161640 1 -- 172.21.15.161:0/1580114759 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6888036930 con 0x7f68980b4260 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f68a03c7640 1 -- 172.21.15.161:0/1580114759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b4260 msgr2=0x7f689815b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:46.183 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f68a03c7640 1 --2- 172.21.15.161:0/1580114759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b4260 0x7f689815b560 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f688800bb10 tx=0x7f688802e150 comp rx=0 tx=0).stop 2024-03-23T14:37:46.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f68a03c7640 1 -- 172.21.15.161:0/1580114759 shutdown_connections 2024-03-23T14:37:46.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f68a03c7640 1 --2- 172.21.15.161:0/1580114759 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b3610 0x7f68980b3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f68a03c7640 1 --2- 172.21.15.161:0/1580114759 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b4260 0x7f689815b560 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f68a03c7640 1 -- 172.21.15.161:0/1580114759 >> 172.21.15.161:0/1580114759 conn(0x7f68980b17f0 msgr2=0x7f68980b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:46.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f68a03c7640 1 -- 172.21.15.161:0/1580114759 shutdown_connections 2024-03-23T14:37:46.184 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.185+0000 7f68a03c7640 1 -- 172.21.15.161:0/1580114759 wait complete. 2024-03-23T14:37:46.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.186+0000 7f68a03c7640 1 Processor -- start 2024-03-23T14:37:46.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.186+0000 7f68a03c7640 1 -- start start 2024-03-23T14:37:46.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f68a03c7640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f689806eb30 0x7f689814a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:46.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f68a03c7640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b3610 0x7f689814e490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:46.187 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f68a03c7640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b4260 0x7f689814e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f68a03c7640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f68981f1060 con 0x7f68980b3610 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f68a03c7640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f68981f0d80 con 0x7f689806eb30 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f68a03c7640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f689814b0b0 con 0x7f68980b4260 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f689d962640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b3610 0x7f689814e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f689d962640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b3610 0x7f689814e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:38374/0 (socket says 172.21.15.161:38374) 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f689d962640 1 -- 172.21.15.161:0/622952854 learned_addr learned my addr 172.21.15.161:0/622952854 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f689d962640 1 -- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b4260 msgr2=0x7f689814e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f689d962640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b4260 0x7f689814e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.188 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f689d962640 1 -- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f689806eb30 msgr2=0x7f689814a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.187+0000 7f689d962640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f689806eb30 0x7f689814a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.188+0000 7f689d962640 1 -- 172.21.15.161:0/622952854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688800b770 con 0x7f68980b3610 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.188+0000 7f689d962640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b3610 0x7f689814e490 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6888039040 tx=0x7f68880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.188+0000 7f688effd640 1 -- 172.21.15.161:0/622952854 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6888036440 con 0x7f68980b3610 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.188+0000 7f688effd640 1 -- 172.21.15.161:0/622952854 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6888004230 con 0x7f68980b3610 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.188+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f689814b390 con 0x7f68980b3610 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.188+0000 7f688effd640 1 -- 172.21.15.161:0/622952854 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6888034980 con 0x7f68980b3610 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.188+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f689814abf0 con 0x7f68980b3610 2024-03-23T14:37:46.189 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.189+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f689814af10 con 0x7f68980b3610 2024-03-23T14:37:46.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.189+0000 7f688effd640 1 -- 172.21.15.161:0/622952854 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f688803d040 con 0x7f68980b3610 2024-03-23T14:37:46.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.189+0000 7f688effd640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f687c06b810 0x7f687c06dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:37:46.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.190+0000 7f688effd640 1 -- 172.21.15.161:0/622952854 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f68880aeb30 con 0x7f68980b3610 2024-03-23T14:37:46.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.190+0000 7f689e163640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f687c06b810 0x7f687c06dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:37:46.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.190+0000 7f689e163640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f687c06b810 0x7f687c06dcd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6890007fd0 tx=0x7f6890007920 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:37:46.190 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.194+0000 7f688effd640 1 -- 172.21.15.161:0/622952854 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f689814af10 con 0x7f68980b3610 2024-03-23T14:37:46.362 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.367+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f689814ad80 con 0x7f687c06b810 2024-03-23T14:37:46.363 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.368+0000 7f688effd640 1 -- 172.21.15.161:0/622952854 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13489 (secure 0 0 0) 0x7f689814ad80 con 0x7f687c06b810 2024-03-23T14:37:46.364 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:37:46.364 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:37:46.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f687c06b810 msgr2=0x7f687c06dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:46.366 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f687c06b810 0x7f687c06dcd0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f6890007fd0 tx=0x7f6890007920 comp rx=0 tx=0).stop 2024-03-23T14:37:46.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b3610 msgr2=0x7f689814e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:37:46.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b3610 0x7f689814e490 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f6888039040 tx=0x7f68880046c0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 shutdown_connections 2024-03-23T14:37:46.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f687c06b810 0x7f687c06dcd0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.367 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f68980b4260 0x7f689814e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f68980b3610 0x7f689814e490 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 --2- 172.21.15.161:0/622952854 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f689806eb30 0x7f689814a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:37:46.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 >> 172.21.15.161:0/622952854 conn(0x7f68980b17f0 msgr2=0x7f68980b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:37:46.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 shutdown_connections 2024-03-23T14:37:46.368 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:37:46.370+0000 7f68a03c7640 1 -- 172.21.15.161:0/622952854 wait complete. 2024-03-23T14:37:46.376 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":126,"stamp":"2024-03-23T14:37:44.506046+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":124,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":222104,"kb_used_data":5928,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752616,"statfs":{"total":773094113280,"available":772866678784,"internally_reserved":0,"allocated":6070272,"data_stored":3204414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":18,"apply_latency_ms":18,"commit_latency_ns":18000000,"apply_latency_ns":18000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-17096,"num_objects":-22,"num_object_clones":0,"num_object_copies":-44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-71,"num_write_kb":-42,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001572"},"pg_stats":[{"pgid":"1.0","version":"16'117","reported_seq":226,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-23T14:37:40.006904+0000","last_change":"2024-03-23T14:35:02.254738+0000","last_active":"2024-03-23T14:37:40.006904+0000","last_peered":"2024-03-23T14:37:40.006904+0000","last_clean":"2024-03-23T14:37:40.006904+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:37:40.006904+0000","last_undegraded":"2024-03-23T14:37:40.006904+0000","last_fullsized":"2024-03-23T14:37:40.006904+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-23T14:35:00.680846+0000","last_clean_scrub_stamp":"2024-03-23T14:35:00.680846+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:40:58.633001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574884,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343656,"statfs":{"total":96636764160,"available":96607903744,"internally_reserved":0,"allocated":1191936,"data_stored":839154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574884,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574883,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574883,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27604,"kb_used_data":584,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344236,"statfs":{"total":96636764160,"available":96608497664,"internally_reserved":0,"allocated":598016,"data_stored":248786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574884,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574885,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574884,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28200,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343640,"statfs":{"total":96636764160,"available":96607887360,"internally_reserved":0,"allocated":1208320,"data_stored":844719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:37:46.379 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-03-23T14:35:00.680846+0000 time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=14, tm_min=35, tm_sec=0, tm_wday=5, tm_yday=83, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=3, tm_mday=23, tm_hour=14, tm_min=37, tm_sec=41, tm_wday=5, tm_yday=83, tm_isdst=0) 2024-03-23T14:37:46.379 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-03-23T14:38:06.380 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-23T14:38:06.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.497+0000 7efeca1c2640 1 Processor -- start 2024-03-23T14:38:06.493 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.497+0000 7efeca1c2640 1 -- start start 2024-03-23T14:38:06.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efeca1c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec4145080 0x7efec4072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efeca1c2640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efec41601f0 0x7efec415c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efeca1c2640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efec40cf4f0 con 0x7efec41607d0 2024-03-23T14:38:06.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efeca1c2640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7efec4071010 con 0x7efec41601f0 2024-03-23T14:38:06.496 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efeca1c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efec40cef30 con 0x7efec4145080 2024-03-23T14:38:06.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efec3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efec41601f0 0x7efec415c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efec3fff640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7efec41601f0 0x7efec415c4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35884/0 (socket says 172.21.15.161:35884) 2024-03-23T14:38:06.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efec3fff640 1 -- 172.21.15.161:0/1271231176 learned_addr learned my addr 172.21.15.161:0/1271231176 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:38:06.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efec37fe640 1 --2- 172.21.15.161:0/1271231176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec4145080 0x7efec4072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efec3fff640 1 -- 172.21.15.161:0/1271231176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec4145080 msgr2=0x7efec4072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:38:06.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efec3fff640 1 --2- 172.21.15.161:0/1271231176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec4145080 0x7efec4072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.497 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.498+0000 7efec3fff640 1 -- 172.21.15.161:0/1271231176 >> v1:172.21.15.183:6789/0 conn(0x7efec41607d0 legacy=0x7efec415fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.499+0000 7efec3fff640 1 -- 172.21.15.161:0/1271231176 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efec40cf210 con 0x7efec41601f0 2024-03-23T14:38:06.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.499+0000 7efec3fff640 1 --2- 172.21.15.161:0/1271231176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41601f0 0x7efec415c4f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7efebc009e50 tx=0x7efebc02bed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cada23bfc35cb46b server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:06.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.499+0000 7efec27fc640 1 -- 172.21.15.161:0/1271231176 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc02ca70 con 0x7efec41601f0 2024-03-23T14:38:06.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.499+0000 7efec27fc640 1 -- 172.21.15.161:0/1271231176 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efebc02cbd0 con 0x7efec41601f0 2024-03-23T14:38:06.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.499+0000 7efec27fc640 1 -- 172.21.15.161:0/1271231176 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc0343f0 con 0x7efec41601f0 2024-03-23T14:38:06.498 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.500+0000 7efeca1c2640 1 -- 172.21.15.161:0/1271231176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41601f0 msgr2=0x7efec415c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.500+0000 7efeca1c2640 1 --2- 172.21.15.161:0/1271231176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41601f0 0x7efec415c4f0 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7efebc009e50 tx=0x7efebc02bed0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.500+0000 7efeca1c2640 1 -- 172.21.15.161:0/1271231176 shutdown_connections 2024-03-23T14:38:06.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.500+0000 7efeca1c2640 1 --2- 172.21.15.161:0/1271231176 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41601f0 0x7efec415c4f0 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.500+0000 7efeca1c2640 1 --2- 172.21.15.161:0/1271231176 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec4145080 0x7efec4072a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.500+0000 7efeca1c2640 1 -- 172.21.15.161:0/1271231176 >> 172.21.15.161:0/1271231176 conn(0x7efec40b17f0 msgr2=0x7efec40b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:38:06.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.500+0000 7efeca1c2640 1 -- 172.21.15.161:0/1271231176 shutdown_connections 2024-03-23T14:38:06.499 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.500+0000 7efeca1c2640 1 -- 172.21.15.161:0/1271231176 wait complete. 2024-03-23T14:38:06.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.501+0000 7efeca1c2640 1 Processor -- start 2024-03-23T14:38:06.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.501+0000 7efeca1c2640 1 -- start start 2024-03-23T14:38:06.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efeca1c2640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec4145080 0x7efec4153a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efeca1c2640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec41601f0 0x7efec41f2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.502 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efeca1c2640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41607d0 0x7efec41f3480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efeca1c2640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7efec413b130 con 0x7efec41607d0 2024-03-23T14:38:06.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efeca1c2640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7efec413d3e0 con 0x7efec4145080 2024-03-23T14:38:06.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efeca1c2640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7efec413c4f0 con 0x7efec41601f0 2024-03-23T14:38:06.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efec3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41607d0 0x7efec41f3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efec3fff640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41607d0 0x7efec41f3480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35886/0 (socket says 172.21.15.161:35886) 2024-03-23T14:38:06.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efec3fff640 1 -- 172.21.15.161:0/3848761129 learned_addr learned my addr 172.21.15.161:0/3848761129 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:38:06.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efec37fe640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec4145080 0x7efec4153a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.503 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.502+0000 7efec2ffd640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec41601f0 0x7efec41f2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efec3fff640 1 -- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec41601f0 msgr2=0x7efec41f2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efec3fff640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec41601f0 0x7efec41f2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efec3fff640 1 -- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec4145080 msgr2=0x7efec4153a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efec3fff640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec4145080 0x7efec4153a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efec3fff640 1 -- 172.21.15.161:0/3848761129 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efebc009b30 con 0x7efec41607d0 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efec2ffd640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec41601f0 0x7efec41f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efec37fe640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec4145080 0x7efec4153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efec3fff640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41607d0 0x7efec41f3480 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7efebc009e50 tx=0x7efebc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:06.504 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efeabfff640 1 -- 172.21.15.161:0/3848761129 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc009510 con 0x7efec41607d0 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efeabfff640 1 -- 172.21.15.161:0/3848761129 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efebc03e450 con 0x7efec41607d0 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efec413c7d0 con 0x7efec41607d0 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efeabfff640 1 -- 172.21.15.161:0/3848761129 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efebc03e800 con 0x7efec41607d0 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.503+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efec413cd90 con 0x7efec41607d0 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.504+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efec40cd810 con 0x7efec41607d0 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.504+0000 7efeabfff640 1 -- 172.21.15.161:0/3848761129 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7efebc03e960 con 0x7efec41607d0 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.505+0000 7efeabfff640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb006b810 0x7efeb006dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.505+0000 7efeabfff640 1 -- 172.21.15.161:0/3848761129 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7efebc0acbe0 con 0x7efec41607d0 2024-03-23T14:38:06.505 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.505+0000 7efec37fe640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb006b810 0x7efeb006dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.506+0000 7efec37fe640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb006b810 0x7efeb006dcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efeb4006fd0 tx=0x7efeb4008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:06.506 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.509+0000 7efeabfff640 1 -- 172.21.15.161:0/3848761129 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7efebc077900 con 0x7efec41607d0 2024-03-23T14:38:06.674 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.678+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 --> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7efec4145460 con 0x7efeb006b810 2024-03-23T14:38:06.675 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.679+0000 7efeabfff640 1 -- 172.21.15.161:0/3848761129 <== mgr.4101 v2:172.21.15.183:6832/3827122778 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+13359 (secure 0 0 0) 0x7efeb4009e10 con 0x7efeb006b810 2024-03-23T14:38:06.675 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:38:06.675 INFO:teuthology.orchestra.run.smithi161.stderr:dumped all 2024-03-23T14:38:06.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb006b810 msgr2=0x7efeb006dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb006b810 0x7efeb006dcd0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efeb4006fd0 tx=0x7efeb4008040 comp rx=0 tx=0).stop 2024-03-23T14:38:06.677 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41607d0 msgr2=0x7efec41f3480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41607d0 0x7efec41f3480 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7efebc009e50 tx=0x7efebc004a40 comp rx=0 tx=0).stop 2024-03-23T14:38:06.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 shutdown_connections 2024-03-23T14:38:06.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7efeb006b810 0x7efeb006dcd0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7efec41607d0 0x7efec41f3480 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.678 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7efec41601f0 0x7efec41f2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 --2- 172.21.15.161:0/3848761129 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7efec4145080 0x7efec4153a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 >> 172.21.15.161:0/3848761129 conn(0x7efec40b17f0 msgr2=0x7efec4142670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:38:06.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 shutdown_connections 2024-03-23T14:38:06.679 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.681+0000 7efeca1c2640 1 -- 172.21.15.161:0/3848761129 wait complete. 2024-03-23T14:38:06.688 INFO:teuthology.orchestra.run.smithi161.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2024-03-23T14:38:06.510504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":2,"kb":754974720,"kb_used":222128,"kb_used_data":5960,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754752592,"statfs":{"total":773094113280,"available":772866654208,"internally_reserved":0,"allocated":6103040,"data_stored":3215544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12750,"internal_metadata":221236786},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002569"},"pg_stats":[{"pgid":"1.0","version":"51'119","reported_seq":237,"reported_epoch":51,"state":"active+clean","last_fresh":"2024-03-23T14:37:46.514959+0000","last_change":"2024-03-23T14:37:46.515025+0000","last_active":"2024-03-23T14:37:46.514959+0000","last_peered":"2024-03-23T14:37:46.514959+0000","last_clean":"2024-03-23T14:37:46.514959+0000","last_became_active":"2024-03-23T14:35:02.253749+0000","last_became_peered":"2024-03-23T14:35:02.253749+0000","last_unstale":"2024-03-23T14:37:46.514959+0000","last_undegraded":"2024-03-23T14:37:46.514959+0000","last_fullsized":"2024-03-23T14:37:46.514959+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"51'119","last_scrub_stamp":"2024-03-23T14:37:46.514797+0000","last_deep_scrub":"51'119","last_deep_scrub_stamp":"2024-03-23T14:37:46.514797+0000","last_clean_scrub_stamp":"2024-03-23T14:37:46.514797+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-03-24T14:39:38.907187+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1208320,"data_stored":844719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":13,"seq":55834574888,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27620,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344220,"statfs":{"total":96636764160,"available":96608481280,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27624,"kb_used_data":600,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344216,"statfs":{"total":96636764160,"available":96608477184,"internally_reserved":0,"allocated":614400,"data_stored":254351,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":13,"seq":55834574889,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":28196,"kb_used_data":1180,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343644,"statfs":{"total":96636764160,"available":96607891456,"internally_reserved":0,"allocated":1208320,"data_stored":844719,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-23T14:38:06.689 DEBUG:teuthology.orchestra.run.smithi161:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-03-23T14:38:06.789 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.793+0000 7fb16c040640 1 Processor -- start 2024-03-23T14:38:06.790 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.794+0000 7fb16c040640 1 -- start start 2024-03-23T14:38:06.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.794+0000 7fb16c040640 1 --2- >> v2:172.21.15.161:3300/0 conn(0x7fb1640b43f0 0x7fb16415b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.794+0000 7fb16c040640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7fb1640d05f0 0x7fb16415b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.794+0000 7fb16c040640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb16415c1f0 con 0x7fb1640b4010 2024-03-23T14:38:06.791 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.794+0000 7fb16c040640 1 -- --> v2:172.21.15.161:3300/0 -- mon_getmap magic: 0 -- 0x7fb16406ef20 con 0x7fb1640b43f0 2024-03-23T14:38:06.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.794+0000 7fb16c040640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7fb16415f730 con 0x7fb1640d05f0 2024-03-23T14:38:06.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb169ddc640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7fb1640b4010 0x7fb1640cfee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60854/0 (socket says 172.21.15.161:60854) 2024-03-23T14:38:06.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb169ddc640 1 -- 172.21.15.161:0/1550692764 learned_addr learned my addr 172.21.15.161:0/1550692764 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:38:06.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb1695db640 1 --2- 172.21.15.161:0/1550692764 >> v2:172.21.15.161:3300/0 conn(0x7fb1640b43f0 0x7fb16415b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb1695db640 1 -- 172.21.15.161:0/1550692764 >> v2:172.21.15.183:3300/0 conn(0x7fb1640d05f0 msgr2=0x7fb16415b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:38:06.792 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb16a5dd640 1 --2- 172.21.15.161:0/1550692764 >> v2:172.21.15.183:3300/0 conn(0x7fb1640d05f0 0x7fb16415b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb1695db640 1 --2- 172.21.15.161:0/1550692764 >> v2:172.21.15.183:3300/0 conn(0x7fb1640d05f0 0x7fb16415b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb1695db640 1 -- 172.21.15.161:0/1550692764 >> v1:172.21.15.161:6789/0 conn(0x7fb1640b4010 legacy=0x7fb1640cfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb168dda640 1 -- 172.21.15.161:0/1550692764 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 958479389 0 0) 0x7fb16415c1f0 con 0x7fb1640b4010 2024-03-23T14:38:06.793 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.795+0000 7fb1695db640 1 -- 172.21.15.161:0/1550692764 --> v2:172.21.15.161:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb16415e970 con 0x7fb1640b43f0 2024-03-23T14:38:06.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb1695db640 1 --2- 172.21.15.161:0/1550692764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b43f0 0x7fb16415b480 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb1540097a0 tx=0x7fb15402b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24eea3384a817f5e server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:06.797 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb168dda640 1 -- 172.21.15.161:0/1550692764 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb15402e070 con 0x7fb1640b43f0 2024-03-23T14:38:06.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb168dda640 1 -- 172.21.15.161:0/1550692764 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1540315a0 con 0x7fb1640b43f0 2024-03-23T14:38:06.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb168dda640 1 -- 172.21.15.161:0/1550692764 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb154031930 con 0x7fb1640b43f0 2024-03-23T14:38:06.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb16c040640 1 -- 172.21.15.161:0/1550692764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b43f0 msgr2=0x7fb16415b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb16c040640 1 --2- 172.21.15.161:0/1550692764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b43f0 0x7fb16415b480 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fb1540097a0 tx=0x7fb15402b840 comp rx=0 tx=0).stop 2024-03-23T14:38:06.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb16c040640 1 -- 172.21.15.161:0/1550692764 shutdown_connections 2024-03-23T14:38:06.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb16c040640 1 --2- 172.21.15.161:0/1550692764 >> v2:172.21.15.183:3300/0 conn(0x7fb1640d05f0 0x7fb16415b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.798 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb16c040640 1 --2- 172.21.15.161:0/1550692764 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b43f0 0x7fb16415b480 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb16c040640 1 -- 172.21.15.161:0/1550692764 >> 172.21.15.161:0/1550692764 conn(0x7fb16406b4a0 msgr2=0x7fb16406b8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:38:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb16c040640 1 -- 172.21.15.161:0/1550692764 shutdown_connections 2024-03-23T14:38:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.796+0000 7fb16c040640 1 -- 172.21.15.161:0/1550692764 wait complete. 2024-03-23T14:38:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb16c040640 1 Processor -- start 2024-03-23T14:38:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb16c040640 1 -- start start 2024-03-23T14:38:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb16c040640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b4010 0x7fb164157930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.799 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb16c040640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1640b43f0 0x7fb164149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb16c040640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1640d05f0 0x7fb1641520a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb16c040640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7fb1641587c0 con 0x7fb1640b4010 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb16c040640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7fb164149a50 con 0x7fb1640d05f0 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb16c040640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7fb164157f20 con 0x7fb1640b43f0 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb169ddc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b4010 0x7fb164157930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb169ddc640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b4010 0x7fb164157930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35894/0 (socket says 172.21.15.161:35894) 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.798+0000 7fb169ddc640 1 -- 172.21.15.161:0/2172552093 learned_addr learned my addr 172.21.15.161:0/2172552093 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb16a5dd640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1640d05f0 0x7fb1641520a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.800 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb1695db640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1640b43f0 0x7fb164149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb169ddc640 1 -- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1640b43f0 msgr2=0x7fb164149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb169ddc640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1640b43f0 0x7fb164149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb169ddc640 1 -- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1640d05f0 msgr2=0x7fb1641520a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb169ddc640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1640d05f0 0x7fb1641520a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb169ddc640 1 -- 172.21.15.161:0/2172552093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb154009400 con 0x7fb1640b4010 2024-03-23T14:38:06.801 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb16a5dd640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1640d05f0 0x7fb1641520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:38:06.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb169ddc640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b4010 0x7fb164157930 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb16000c4d0 tx=0x7fb16000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:06.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb15a7fc640 1 -- 172.21.15.161:0/2172552093 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1600074e0 con 0x7fb1640b4010 2024-03-23T14:38:06.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb15a7fc640 1 -- 172.21.15.161:0/2172552093 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb160007660 con 0x7fb1640b4010 2024-03-23T14:38:06.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb15a7fc640 1 -- 172.21.15.161:0/2172552093 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb160004a20 con 0x7fb1640b4010 2024-03-23T14:38:06.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb164158200 con 0x7fb1640b4010 2024-03-23T14:38:06.802 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.799+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1640ccff0 con 0x7fb1640b4010 2024-03-23T14:38:06.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.800+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1640cce50 con 0x7fb1640b4010 2024-03-23T14:38:06.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.800+0000 7fb15a7fc640 1 -- 172.21.15.161:0/2172552093 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7fb160004b80 con 0x7fb1640b4010 2024-03-23T14:38:06.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.801+0000 7fb15a7fc640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb14806b810 0x7fb14806dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:06.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.801+0000 7fb15a7fc640 1 -- 172.21.15.161:0/2172552093 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7fb160089450 con 0x7fb1640b4010 2024-03-23T14:38:06.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.801+0000 7fb1695db640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb14806b810 0x7fb14806dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:06.803 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.802+0000 7fb1695db640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb14806b810 0x7fb14806dcd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb154036000 tx=0x7fb154008030 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:06.804 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.805+0000 7fb15a7fc640 1 -- 172.21.15.161:0/2172552093 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7fb1640cce50 con 0x7fb1640b4010 2024-03-23T14:38:06.972 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.976+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7fb1640cce50 con 0x7fb1640b4010 2024-03-23T14:38:06.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.981+0000 7fb15a7fc640 1 -- 172.21.15.161:0/2172552093 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v14)=0 v14) ==== 123+0+0 (secure 0 0 0) 0x7fb1640cce50 con 0x7fb1640b4010 2024-03-23T14:38:06.977 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.982+0000 7fb15a7fc640 1 -- 172.21.15.161:0/2172552093 <== mon.0 v2:172.21.15.161:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fb160054170 con 0x7fb1640b4010 2024-03-23T14:38:06.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb14806b810 msgr2=0x7fb14806dcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.980 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb14806b810 0x7fb14806dcd0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fb154036000 tx=0x7fb154008030 comp rx=0 tx=0).stop 2024-03-23T14:38:06.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b4010 msgr2=0x7fb164157930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:06.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b4010 0x7fb164157930 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7fb16000c4d0 tx=0x7fb16000c990 comp rx=0 tx=0).stop 2024-03-23T14:38:06.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 shutdown_connections 2024-03-23T14:38:06.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7fb14806b810 0x7fb14806dcd0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7fb1640d05f0 0x7fb1641520a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7fb1640b43f0 0x7fb164149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 --2- 172.21.15.161:0/2172552093 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7fb1640b4010 0x7fb164157930 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:06.981 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 >> 172.21.15.161:0/2172552093 conn(0x7fb16406b4a0 msgr2=0x7fb164152e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:38:06.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 shutdown_connections 2024-03-23T14:38:06.982 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:06.984+0000 7fb16c040640 1 -- 172.21.15.161:0/2172552093 wait complete. 2024-03-23T14:38:06.991 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-23T14:38:06.991 DEBUG:teuthology.orchestra.run.smithi161:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-23T14:38:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.096+0000 7f667bf81640 1 Processor -- start 2024-03-23T14:38:07.092 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.096+0000 7f667bf81640 1 -- start start 2024-03-23T14:38:07.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f667bf81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674145080 0x7f6674072a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f667bf81640 1 --2- >> v2:172.21.15.183:3300/0 conn(0x7f66741601f0 0x7f667415c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f667bf81640 1 -- --> v1:172.21.15.161:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66740cf4f0 con 0x7f66741607d0 2024-03-23T14:38:07.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f667bf81640 1 -- --> v2:172.21.15.183:3300/0 -- mon_getmap magic: 0 -- 0x7f6674071010 con 0x7f66741601f0 2024-03-23T14:38:07.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f667bf81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f66740cef30 con 0x7f6674145080 2024-03-23T14:38:07.096 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f667951c640 1 --1- >> v1:172.21.15.161:6789/0 conn(0x7f66741607d0 0x7f667415fae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.161:6789/0 says I am v1:172.21.15.161:60862/0 (socket says 172.21.15.161:60862) 2024-03-23T14:38:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f667951c640 1 -- 172.21.15.161:0/1447690910 learned_addr learned my addr 172.21.15.161:0/1447690910 (peer_addr_for_me v1:172.21.15.161:0/0) 2024-03-23T14:38:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f667a51e640 1 --2- 172.21.15.161:0/1447690910 >> v2:172.21.15.183:3300/0 conn(0x7f66741601f0 0x7f667415c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f6679d1d640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674145080 0x7f6674072a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 <== mon.0 v1:172.21.15.161:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252650353 0 0) 0x7f66740cf4f0 con 0x7f66741607d0 2024-03-23T14:38:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 --> v1:172.21.15.161:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6660003520 con 0x7f66741607d0 2024-03-23T14:38:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 <== mon.0 v1:172.21.15.161:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1758438892 0 0) 0x7f6660003520 con 0x7f66741607d0 2024-03-23T14:38:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674145080 msgr2=0x7f6674072a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:38:07.097 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.097+0000 7f6678d1b640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674145080 0x7f6674072a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 >> v2:172.21.15.183:3300/0 conn(0x7f66741601f0 msgr2=0x7f667415c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:38:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 --2- 172.21.15.161:0/1447690910 >> v2:172.21.15.183:3300/0 conn(0x7f66741601f0 0x7f667415c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 --> v1:172.21.15.161:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66740cf210 con 0x7f66741607d0 2024-03-23T14:38:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 <== mon.0 v1:172.21.15.161:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 2420109360 0 0) 0x7f6664001670 con 0x7f66741607d0 2024-03-23T14:38:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 >> v1:172.21.15.161:6789/0 conn(0x7f66741607d0 legacy=0x7f667415fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6660003970 0x7f6660003d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674145080 0x7f66600047c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.098 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6660004d00 0x7f6660005180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f6674071010 con 0x7f6660004d00 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f66740cef30 con 0x7f6660003970 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f666000b980 con 0x7f6674145080 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f667a51e640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6660004d00 0x7f6660005180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f6679d1d640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6660003970 0x7f6660003d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f667951c640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674145080 0x7f66600047c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f667a51e640 1 -- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674145080 msgr2=0x7f66600047c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f667a51e640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674145080 0x7f66600047c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.099 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f667a51e640 1 -- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6660003970 msgr2=0x7f6660003d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-23T14:38:07.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f667a51e640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f6660003970 0x7f6660003d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.098+0000 7f667a51e640 1 -- 172.21.15.161:0/1447690910 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6660003520 con 0x7f6660004d00 2024-03-23T14:38:07.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f667a51e640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6660004d00 0x7f6660005180 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f666c009fc0 tx=0x7f66600057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d428504774bf3547 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:07.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c02cc60 con 0x7f6660004d00 2024-03-23T14:38:07.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f666c00f040 con 0x7f6660004d00 2024-03-23T14:38:07.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f6678d1b640 1 -- 172.21.15.161:0/1447690910 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c034440 con 0x7f6660004d00 2024-03-23T14:38:07.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f667bf81640 1 -- 172.21.15.161:0/1447690910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6660004d00 msgr2=0x7f6660005180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:07.100 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f667bf81640 1 --2- 172.21.15.161:0/1447690910 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f6660004d00 0x7f6660005180 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f666c009fc0 tx=0x7f66600057f0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f667a51e640 1 -- 172.21.15.161:0/1447690910 reap_dead start 2024-03-23T14:38:07.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f667bf81640 1 -- 172.21.15.161:0/1447690910 shutdown_connections 2024-03-23T14:38:07.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.099+0000 7f667bf81640 1 -- 172.21.15.161:0/1447690910 >> 172.21.15.161:0/1447690910 conn(0x7f66740b17f0 msgr2=0x7f66740b1bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:38:07.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.100+0000 7f667bf81640 1 -- 172.21.15.161:0/1447690910 shutdown_connections 2024-03-23T14:38:07.101 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.100+0000 7f667bf81640 1 -- 172.21.15.161:0/1447690910 wait complete. 2024-03-23T14:38:07.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.101+0000 7f667bf81640 1 Processor -- start 2024-03-23T14:38:07.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.101+0000 7f667bf81640 1 -- start start 2024-03-23T14:38:07.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.101+0000 7f667bf81640 1 --2- >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66741344b0 0x7f667413ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.101+0000 7f667bf81640 1 --2- >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674134890 0x7f66740aa1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.101+0000 7f667bf81640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66740b3570 0x7f66740b3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.104 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.101+0000 7f667bf81640 1 -- --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_getmap magic: 0 -- 0x7f66740b3f60 con 0x7f66740b3570 2024-03-23T14:38:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.101+0000 7f667bf81640 1 -- --> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] -- mon_getmap magic: 0 -- 0x7f667413fe40 con 0x7f66741344b0 2024-03-23T14:38:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.101+0000 7f667bf81640 1 -- --> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] -- mon_getmap magic: 0 -- 0x7f667413f230 con 0x7f6674134890 2024-03-23T14:38:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66740b3570 0x7f66740b3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 --2- >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66740b3570 0x7f66740b3a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.161:3300/0 says I am v2:172.21.15.161:35914/0 (socket says 172.21.15.161:35914) 2024-03-23T14:38:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 -- 172.21.15.161:0/429783631 learned_addr learned my addr 172.21.15.161:0/429783631 (peer_addr_for_me v2:172.21.15.161:0/0) 2024-03-23T14:38:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f6679d1d640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66741344b0 0x7f667413ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667951c640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674134890 0x7f66740aa1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.105 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 -- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674134890 msgr2=0x7f66740aa1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674134890 0x7f66740aa1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 -- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66741344b0 msgr2=0x7f667413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66741344b0 0x7f667413ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 -- 172.21.15.161:0/429783631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f667413f510 con 0x7f66740b3570 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f6679d1d640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66741344b0 0x7f667413ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667a51e640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66740b3570 0x7f66740b3a20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f666c00e230 tx=0x7f666c009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f666a7fc640 1 -- 172.21.15.161:0/429783631 <== mon.0 v2:172.21.15.161:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c034d50 con 0x7f66740b3570 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f666a7fc640 1 -- 172.21.15.161:0/429783631 <== mon.0 v2:172.21.15.161:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f666c003990 con 0x7f66740b3570 2024-03-23T14:38:07.106 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f666a7fc640 1 -- 172.21.15.161:0/429783631 <== mon.0 v2:172.21.15.161:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f666c003d60 con 0x7f66740b3570 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f667413f7f0 con 0x7f66740b3570 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.102+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66740aa730 con 0x7f66740b3570 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.103+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f667406cb70 con 0x7f66740b3570 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.103+0000 7f666a7fc640 1 -- 172.21.15.161:0/429783631 <== mon.0 v2:172.21.15.161:3300/0 4 ==== mgrmap(e 5) ==== 89780+0+0 (secure 0 0 0) 0x7f666c004030 con 0x7f66740b3570 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.104+0000 7f666a7fc640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b840 0x7f665806dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.104+0000 7f666a7fc640 1 -- 172.21.15.161:0/429783631 <== mon.0 v2:172.21.15.161:3300/0 5 ==== osd_map(51..51 src has 1..51) ==== 4943+0+0 (secure 0 0 0) 0x7f666c0ab6d0 con 0x7f66740b3570 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.104+0000 7f6679d1d640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b840 0x7f665806dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.105+0000 7f6679d1d640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b840 0x7f665806dd00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6670009bc0 tx=0x7f6670008040 comp rx=0 tx=0).ready entity=mgr.4101 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-23T14:38:07.107 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.108+0000 7f666a7fc640 1 -- 172.21.15.161:0/429783631 <== mon.0 v2:172.21.15.161:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+183808 (secure 0 0 0) 0x7f666c076300 con 0x7f66740b3570 2024-03-23T14:38:07.312 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.316+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 --> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f66740735e0 con 0x7f66740b3570 2024-03-23T14:38:07.313 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.317+0000 7f666a7fc640 1 -- 172.21.15.161:0/429783631 <== mon.0 v2:172.21.15.161:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+3881 (secure 0 0 0) 0x7f666c078230 con 0x7f66740b3570 2024-03-23T14:38:07.313 INFO:teuthology.orchestra.run.smithi161.stdout: 2024-03-23T14:38:07.313 INFO:teuthology.orchestra.run.smithi161.stdout:{"epoch":25,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":5393,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6836/2496234294","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6835","nonce":2496234294},{"type":"v1","addr":"172.21.15.183:6836","nonce":2496234294}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5396,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.183:6837/2645821131","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.183:6834","nonce":2645821131},{"type":"v1","addr":"172.21.15.183:6837","nonce":2645821131}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":20},{"gid":5922,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6834/2183619206","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6832","nonce":2183619206},{"type":"v1","addr":"172.21.15.161:6834","nonce":2183619206}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":19},{"gid":6372,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.161:6835/1071965002","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.161:6833","nonce":1071965002},{"type":"v1","addr":"172.21.15.161:6835","nonce":1071965002}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":25}],"filesystems":[]} 2024-03-23T14:38:07.314 INFO:teuthology.orchestra.run.smithi161.stderr:dumped fsmap epoch 25 2024-03-23T14:38:07.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b840 msgr2=0x7f665806dd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:07.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b840 0x7f665806dd00 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f6670009bc0 tx=0x7f6670008040 comp rx=0 tx=0).stop 2024-03-23T14:38:07.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66740b3570 msgr2=0x7f66740b3a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-23T14:38:07.315 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66740b3570 0x7f66740b3a20 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f666c00e230 tx=0x7f666c009750 comp rx=0 tx=0).stop 2024-03-23T14:38:07.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 shutdown_connections 2024-03-23T14:38:07.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:6832/3827122778,v1:172.21.15.183:6833/3827122778] conn(0x7f665806b840 0x7f665806dd00 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.161:3300/0,v1:172.21.15.161:6789/0] conn(0x7f66740b3570 0x7f66740b3a20 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3301/0,v1:172.21.15.183:6790/0] conn(0x7f6674134890 0x7f66740aa1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 --2- 172.21.15.161:0/429783631 >> [v2:172.21.15.183:3300/0,v1:172.21.15.183:6789/0] conn(0x7f66741344b0 0x7f667413ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-23T14:38:07.316 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 >> 172.21.15.161:0/429783631 conn(0x7f66740b17f0 msgr2=0x7f66740af710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-23T14:38:07.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 shutdown_connections 2024-03-23T14:38:07.317 INFO:teuthology.orchestra.run.smithi161.stderr:2024-03-23T14:38:07.319+0000 7f667bf81640 1 -- 172.21.15.161:0/429783631 wait complete. 2024-03-23T14:38:07.326 DEBUG:tasks.cephfs.filesystem:already dead... 2024-03-23T14:38:07.326 INFO:teuthology.misc:Shutting down mds daemons... 2024-03-23T14:38:07.326 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-03-23T14:38:07.326 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.336 INFO:tasks.ceph.mds.a:Stopped 2024-03-23T14:38:07.336 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-03-23T14:38:07.336 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.385 INFO:tasks.ceph.mds.c:Stopped 2024-03-23T14:38:07.385 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-03-23T14:38:07.386 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.398 INFO:tasks.ceph.mds.b:Stopped 2024-03-23T14:38:07.398 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-03-23T14:38:07.398 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.449 INFO:tasks.ceph.mds.d:Stopped 2024-03-23T14:38:07.450 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-23T14:38:07.450 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-23T14:38:07.450 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.481 INFO:tasks.ceph.osd.0:Stopped 2024-03-23T14:38:07.482 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-23T14:38:07.482 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.553 INFO:tasks.ceph.osd.1:Stopped 2024-03-23T14:38:07.553 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-23T14:38:07.553 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.621 INFO:tasks.ceph.osd.2:Stopped 2024-03-23T14:38:07.622 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-23T14:38:07.622 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.691 INFO:tasks.ceph.osd.3:Stopped 2024-03-23T14:38:07.691 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-23T14:38:07.691 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.735 INFO:tasks.ceph.osd.4:Stopped 2024-03-23T14:38:07.736 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-23T14:38:07.736 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.772 INFO:tasks.ceph.osd.5:Stopped 2024-03-23T14:38:07.772 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-23T14:38:07.772 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.827 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:38:07.829 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:38:07.864 INFO:tasks.ceph.osd.6:Stopped 2024-03-23T14:38:07.864 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-23T14:38:07.864 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.918 INFO:tasks.ceph.osd.7:Stopped 2024-03-23T14:38:07.918 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-23T14:38:07.919 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-23T14:38:07.919 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.943 INFO:tasks.ceph.mgr.y:Stopped 2024-03-23T14:38:07.943 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-23T14:38:07.943 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:07.988 INFO:tasks.ceph.mgr.x:Stopped 2024-03-23T14:38:07.988 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-23T14:38:07.989 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-23T14:38:07.989 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:08.000 INFO:tasks.ceph.mon.a:Stopped 2024-03-23T14:38:08.000 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-23T14:38:08.000 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:08.040 INFO:tasks.ceph.mon.b:Stopped 2024-03-23T14:38:08.040 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-23T14:38:08.040 INFO:teuthology.orchestra.run:waiting for 300 2024-03-23T14:38:08.093 INFO:tasks.ceph.mon.c:Stopped 2024-03-23T14:38:08.094 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-23T14:38:08.094 DEBUG:teuthology.orchestra.run.smithi161:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | head -n 1 2024-03-23T14:38:08.131 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:38:08.131 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-23T14:38:08.312 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:38:08.312 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-23T14:38:08.401 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:38:08.401 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-23T14:38:08.462 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:38:08.462 DEBUG:teuthology.orchestra.run.smithi161:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-23T14:38:08.508 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:38:08.508 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-23T14:38:08.693 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:38:08.694 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-23T14:38:08.776 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:38:08.776 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-23T14:38:08.857 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:38:08.858 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-23T14:38:08.913 INFO:tasks.ceph:Archiving mon data... 2024-03-23T14:38:08.915 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/data/mon.a.tgz 2024-03-23T14:38:08.915 DEBUG:teuthology.orchestra.run.smithi161:> mktemp 2024-03-23T14:38:08.931 INFO:teuthology.orchestra.run.smithi161.stdout:/tmp/tmp.kYia3q5nkJ 2024-03-23T14:38:08.931 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kYia3q5nkJ 2024-03-23T14:38:09.074 DEBUG:teuthology.orchestra.run.smithi161:> sudo chmod 0666 /tmp/tmp.kYia3q5nkJ 2024-03-23T14:38:09.160 DEBUG:teuthology.orchestra.remote:smithi161:/tmp/tmp.kYia3q5nkJ is 336KB 2024-03-23T14:38:09.256 DEBUG:teuthology.orchestra.run.smithi161:> rm -fr /tmp/tmp.kYia3q5nkJ 2024-03-23T14:38:09.270 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/data/mon.b.tgz 2024-03-23T14:38:09.271 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-23T14:38:09.288 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.nPrgNYbOhV 2024-03-23T14:38:09.289 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.nPrgNYbOhV 2024-03-23T14:38:09.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.nPrgNYbOhV 2024-03-23T14:38:09.537 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.nPrgNYbOhV is 445KB 2024-03-23T14:38:09.635 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.nPrgNYbOhV 2024-03-23T14:38:09.649 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/data/mon.c.tgz 2024-03-23T14:38:09.649 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-23T14:38:09.703 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.2Kpvv3OwDk 2024-03-23T14:38:09.703 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.2Kpvv3OwDk 2024-03-23T14:38:09.850 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.2Kpvv3OwDk 2024-03-23T14:38:09.936 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.2Kpvv3OwDk is 445KB 2024-03-23T14:38:10.034 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.2Kpvv3OwDk 2024-03-23T14:38:10.049 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-23T14:38:10.049 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T14:38:10.051 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-23T14:38:10.138 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-23T14:38:10.170 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-23T14:38:10.170 INFO:tasks.ceph:Archiving crash dumps... 2024-03-23T14:38:10.172 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/remote/smithi161/crash 2024-03-23T14:38:10.173 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T14:38:10.209 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/remote/smithi183/crash 2024-03-23T14:38:10.210 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-23T14:38:10.236 INFO:tasks.ceph:Shutting down logrotate 2024-03-23T14:38:10.236 DEBUG:teuthology.orchestra.run.smithi161:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:38:10.254 DEBUG:teuthology.orchestra.run.smithi183:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-03-23T14:38:10.487 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-23T14:38:10.519 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-03-23T14:38:10.546 INFO:tasks.ceph:Compressing logs... 2024-03-23T14:38:10.546 DEBUG:teuthology.orchestra.run.smithi161:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-23T14:38:10.562 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-23T14:38:10.591 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56730.log 2024-03-23T14:38:10.592 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60374.log 2024-03-23T14:38:10.592 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53337.log 2024-03-23T14:38:10.592 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52215.log 2024-03-23T14:38:10.592 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.56730.log.gz 2024-03-23T14:38:10.592 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60374.log.gz 2024-03-23T14:38:10.592 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53337.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44114.log 2024-03-23T14:38:10.592 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53337.log.gz 2024-03-23T14:38:10.592 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43184.log 2024-03-23T14:38:10.639 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37782.log 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37742.log 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.37782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37942.log 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.37742.log: gzip -5 88.3% --verbose -- replaced with /var/log/ceph/ceph-client.admin.37782.log.gz -- 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr: /var/log/ceph/ceph-client.admin.37902.log 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.37742.log.gz 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.37942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.3.44648.log 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.37902.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.37942.log.gz 2024-03-23T14:38:10.640 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-23T14:38:10.641 INFO:teuthology.orchestra.run.smithi183.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.37902.log.gz 2024-03-23T14:38:10.641 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.3.44648.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-03-23T14:38:10.641 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37662.log 2024-03-23T14:38:10.641 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-23T14:38:10.641 INFO:teuthology.orchestra.run.smithi183.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.3.44648.log.gz 2024-03-23T14:38:10.641 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.37662.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-23T14:38:10.641 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.4.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.37662.log.gz 2024-03-23T14:38:10.641 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-23T14:38:10.642 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37702.log 2024-03-23T14:38:10.642 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.1.44192.log 2024-03-23T14:38:10.642 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.37702.log: gzip -5 --verbose -- 93.6% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-23T14:38:10.642 INFO:teuthology.orchestra.run.smithi183.stderr: /var/log/ceph/ceph-client.admin.37862.log 2024-03-23T14:38:10.642 INFO:teuthology.orchestra.run.smithi183.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.37702.log.gz 2024-03-23T14:38:10.642 INFO:teuthology.orchestra.run.smithi183.stderr:gzip/var/log/ceph/ceph-client.1.44192.log: -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-23T14:38:10.642 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.37862.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-23T14:38:10.642 INFO:teuthology.orchestra.run.smithi183.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.37862.log.gz 2024-03-23T14:38:10.643 INFO:teuthology.orchestra.run.smithi183.stderr: 89.4%/var/log/ceph/ceph-mgr.x.log: -- replaced with /var/log/ceph/ceph-client.1.44192.log.gz 2024-03-23T14:38:10.643 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37822.log 2024-03-23T14:38:10.643 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-23T14:38:10.643 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.37822.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.37822.log.gz 2024-03-23T14:38:10.643 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-03-23T14:38:10.643 INFO:teuthology.orchestra.run.smithi183.stderr: 92.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-23T14:38:10.643 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-23T14:38:10.643 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.c.log: 91.3% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-03-23T14:38:10.653 INFO:teuthology.orchestra.run.smithi183.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-03-23T14:38:10.667 INFO:teuthology.orchestra.run.smithi183.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-23T14:38:10.739 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.52215.log.gz 2024-03-23T14:38:10.740 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60806.log 2024-03-23T14:38:10.740 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43184.log: gzip 87.8% -5 --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.44114.log.gz /var/log/ceph/ceph-client.admin.43269.log 2024-03-23T14:38:10.740 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-23T14:38:10.740 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60806.log: 89.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43184.log.gz 2024-03-23T14:38:10.740 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.46463.log 2024-03-23T14:38:10.740 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43269.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60806.log.gz 2024-03-23T14:38:10.740 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43395.log 2024-03-23T14:38:10.741 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46463.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43269.log.gz 2024-03-23T14:38:10.741 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-23T14:38:10.741 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.46463.log.gz 2024-03-23T14:38:10.741 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46699.log 2024-03-23T14:38:10.741 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.43395.log.gz 2024-03-23T14:38:10.741 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50482.log 2024-03-23T14:38:10.741 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46699.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44127.log 2024-03-23T14:38:10.741 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46699.log.gz 2024-03-23T14:38:10.742 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50482.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59637.log 2024-03-23T14:38:10.742 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50482.log.gz 2024-03-23T14:38:10.742 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44127.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52561.log 2024-03-23T14:38:10.742 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53295.log 2024-03-23T14:38:10.742 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44127.log.gz 2024-03-23T14:38:10.742 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59637.log.gz 2024-03-23T14:38:10.742 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44448.log 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52561.log.gz 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53295.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54259.log 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44448.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53295.log.gz 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59949.log 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.44448.log.gz 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54259.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.58938.log 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54259.log.gz 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59763.log 2024-03-23T14:38:10.743 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.0.58938.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59949.log.gz 2024-03-23T14:38:10.744 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54962.log 2024-03-23T14:38:10.744 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59763.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47706.log 2024-03-23T14:38:10.744 INFO:teuthology.orchestra.run.smithi161.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.59763.log.gz 2024-03-23T14:38:10.744 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54962.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55508.log 2024-03-23T14:38:10.744 INFO:teuthology.orchestra.run.smithi161.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.0.58938.log.gz/var/log/ceph/ceph-client.admin.47706.log: 2024-03-23T14:38:10.744 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54962.log.gz 2024-03-23T14:38:10.744 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48588.log 2024-03-23T14:38:10.745 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47706.log.gz 2024-03-23T14:38:10.745 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44827.log 2024-03-23T14:38:10.745 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55508.log: /var/log/ceph/ceph-client.admin.48588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46573.log 2024-03-23T14:38:10.745 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.55508.log.gz 2024-03-23T14:38:10.745 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48588.log.gz 2024-03-23T14:38:10.745 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44827.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60932.log 2024-03-23T14:38:10.745 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44827.log.gz 2024-03-23T14:38:10.745 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46573.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59992.log 2024-03-23T14:38:10.746 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60932.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46573.log.gz 2024-03-23T14:38:10.746 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54836.log 2024-03-23T14:38:10.746 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60932.log.gz 2024-03-23T14:38:10.746 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50314.log 2024-03-23T14:38:10.746 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54836.log: gzip -5 --verbose -- 88.4% /var/log/ceph/ceph-client.admin.49186.log 2024-03-23T14:38:10.746 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.59992.log.gz 2024-03-23T14:38:10.746 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50314.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54836.log.gz 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55851.log 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50314.log.gz 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49186.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59359.log 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7%/var/log/ceph/ceph-client.admin.55851.log: -- replaced with /var/log/ceph/ceph-client.admin.49186.log.gz 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56189.log 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59359.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.55851.log.gz 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47359.log 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.59359.log.gz 2024-03-23T14:38:10.747 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48052.log 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6%/var/log/ceph/ceph-client.admin.47359.log: -- replaced with /var/log/ceph/ceph-client.admin.56189.log.gz 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51260.log 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48052.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47359.log.gz 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54401.log 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48052.log.gz 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44462.log 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61224.log 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44462.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54401.log.gz 2024-03-23T14:38:10.748 INFO:teuthology.orchestra.run.smithi161.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.51260.log.gz 2024-03-23T14:38:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60764.log 2024-03-23T14:38:10.749 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44462.log.gz 2024-03-23T14:38:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61224.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58600.log 2024-03-23T14:38:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60764.log: 88.2%gzip -- replaced with /var/log/ceph/ceph-client.admin.61224.log.gz 2024-03-23T14:38:10.749 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.54217.log 2024-03-23T14:38:10.749 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58600.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46825.log 2024-03-23T14:38:10.749 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.60764.log.gz 2024-03-23T14:38:10.749 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.58600.log.gz 2024-03-23T14:38:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54217.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47806.log 2024-03-23T14:38:10.750 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.54217.log.gz 2024-03-23T14:38:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46825.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51871.log 2024-03-23T14:38:10.750 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46825.log.gz 2024-03-23T14:38:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58678.log 2024-03-23T14:38:10.750 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47806.log.gz 2024-03-23T14:38:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51871.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50398.log 2024-03-23T14:38:10.750 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58678.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51871.log.gz 2024-03-23T14:38:10.751 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61266.log 2024-03-23T14:38:10.753 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50398.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57713.log 2024-03-23T14:38:10.753 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58678.log.gz 2024-03-23T14:38:10.754 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50398.log.gz 2024-03-23T14:38:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61266.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60094.log 2024-03-23T14:38:10.754 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.61266.log.gz 2024-03-23T14:38:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57713.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49102.log 2024-03-23T14:38:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60094.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.57713.log.gz 2024-03-23T14:38:10.754 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52357.log 2024-03-23T14:38:10.754 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60094.log.gz 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49102.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61058.log 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49102.log.gz 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52357.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46951.log 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61058.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52357.log.gz 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55634.log 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.61058.log.gz 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46951.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56101.log 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.46951.log.gz 2024-03-23T14:38:10.755 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45205.log 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56101.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55634.log.gz 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60637.log 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45205.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.56101.log.gz 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56370.log 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.45205.log.gz 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60637.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43043.log 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56370.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60637.log.gz 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48388.log 2024-03-23T14:38:10.756 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.56370.log.gz 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43043.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48346.log 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43043.log.gz 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47401.log 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48388.log.gz 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52315.log 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48346.log.gz 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47401.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43609.log 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47401.log.gz 2024-03-23T14:38:10.757 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52315.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55046.log 2024-03-23T14:38:10.758 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52315.log.gz 2024-03-23T14:38:10.758 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54485.log 2024-03-23T14:38:10.758 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55046.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56454.log 2024-03-23T14:38:10.758 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43609.log.gz 2024-03-23T14:38:10.758 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54485.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55046.log.gz 2024-03-23T14:38:10.758 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45079.log 2024-03-23T14:38:10.758 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54485.log.gz 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45121.log 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45079.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56454.log.gz 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50145.log 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.45079.log.gz 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45121.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50103.log 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.45121.log.gz 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50145.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47274.log 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50103.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50145.log.gz 2024-03-23T14:38:10.759 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44122.log 2024-03-23T14:38:10.760 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47274.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50103.log.gz 2024-03-23T14:38:10.760 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44460.log 2024-03-23T14:38:10.760 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.47274.log.gz 2024-03-23T14:38:10.760 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44122.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48504.log 2024-03-23T14:38:10.760 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44460.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44122.log.gz 2024-03-23T14:38:10.760 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58516.log 2024-03-23T14:38:10.760 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.44460.log.gz 2024-03-23T14:38:10.760 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48504.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45391.log 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48504.log.gz 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58516.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43227.log 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45391.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58516.log.gz 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46615.log 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.45391.log.gz 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43227.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47316.log 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46615.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43227.log.gz 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52441.log 2024-03-23T14:38:10.761 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.46615.log.gz 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49522.log 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.47316.log.gz 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52047.log 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49522.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.52441.log.gz 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49060.log 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49522.log.gz 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52047.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43735.log 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52047.log.gz 2024-03-23T14:38:10.762 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49060.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60332.log 2024-03-23T14:38:10.763 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4%/var/log/ceph/ceph-client.admin.43735.log: -- replaced with /var/log/ceph/ceph-client.admin.49060.log.gz 2024-03-23T14:38:10.763 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47180.log 2024-03-23T14:38:10.763 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43735.log.gz 2024-03-23T14:38:10.763 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60332.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56412.log 2024-03-23T14:38:10.763 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60332.log.gz 2024-03-23T14:38:10.763 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47180.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37704.log 2024-03-23T14:38:10.763 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56412.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.47180.log.gz 2024-03-23T14:38:10.763 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49312.log 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.56412.log.gz 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.37704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56017.log 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49312.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59553.log 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.37704.log.gz 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7%/var/log/ceph/ceph-client.admin.56017.log: -- replaced with /var/log/ceph/ceph-client.admin.49312.log.gz 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46783.log 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56017.log.gz 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59553.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48262.log 2024-03-23T14:38:10.764 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.59553.log.gz 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55088.log 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48262.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.46783.log.gz 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55004.log 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48262.log.gz 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50272.log 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6%/var/log/ceph/ceph-client.admin.55004.log: -- replaced with /var/log/ceph/ceph-client.admin.55088.log.gz 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50568.log 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50272.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.55004.log.gz 2024-03-23T14:38:10.765 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49396.log 2024-03-23T14:38:10.766 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50272.log.gz 2024-03-23T14:38:10.766 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50568.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60290.log 2024-03-23T14:38:10.766 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50568.log.gz 2024-03-23T14:38:10.766 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49396.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50440.log 2024-03-23T14:38:10.766 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60290.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49396.log.gz 2024-03-23T14:38:10.766 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49774.log 2024-03-23T14:38:10.766 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.60290.log.gz 2024-03-23T14:38:10.766 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50440.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59847.log 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49774.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50440.log.gz 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59721.log 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59847.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.49774.log.gz 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57349.log 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.59847.log.gz 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59721.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59721.log.gz 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57349.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50654.log 2024-03-23T14:38:10.767 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.audit.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57349.log.gz 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53237.log 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50654.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60593.log 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53237.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50654.log.gz 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60890.log 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53237.log.gz 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50924.log 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.60593.log.gz 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59805.log 2024-03-23T14:38:10.768 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60890.log: 89.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-23T14:38:10.769 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50924.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51218.log 2024-03-23T14:38:10.769 INFO:teuthology.orchestra.run.smithi161.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.60890.log.gz 2024-03-23T14:38:10.769 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59805.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50924.log.gz 2024-03-23T14:38:10.769 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47485.log 2024-03-23T14:38:10.769 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4%/var/log/ceph/ceph-client.admin.51218.log: -- replaced with /var/log/ceph/ceph-client.admin.59805.log.gz 2024-03-23T14:38:10.769 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58290.log 2024-03-23T14:38:10.769 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51218.log.gz 2024-03-23T14:38:10.769 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48672.log 2024-03-23T14:38:10.770 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58290.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.47485.log.gz 2024-03-23T14:38:10.770 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48856.log 2024-03-23T14:38:10.770 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48672.log: gzip -5 --verbose 89.1% -- -- replaced with /var/log/ceph/ceph-client.admin.58290.log.gz /var/log/ceph/ceph-client.admin.55466.log 2024-03-23T14:38:10.770 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-23T14:38:10.770 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48672.log.gz 2024-03-23T14:38:10.770 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43819.log 2024-03-23T14:38:10.770 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55466.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48856.log.gz 2024-03-23T14:38:10.770 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55214.log 2024-03-23T14:38:10.771 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2%/var/log/ceph/ceph-client.admin.43819.log: -- replaced with /var/log/ceph/ceph-client.admin.55466.log.gz 2024-03-23T14:38:10.771 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53463.log 2024-03-23T14:38:10.771 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43819.log.gz 2024-03-23T14:38:10.771 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55214.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.52821.log 2024-03-23T14:38:10.771 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53463.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55214.log.gz 2024-03-23T14:38:10.771 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51176.log 2024-03-23T14:38:10.771 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0%/var/log/ceph/ceph-client.0.52821.log: -- replaced with /var/log/ceph/ceph-client.admin.53463.log.gz 2024-03-23T14:38:10.771 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57467.log 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51176.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48898.log 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57467.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51176.log.gz 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57593.log 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56324.log 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.57467.log.gz 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.0.52821.log.gz 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48898.log: /var/log/ceph/ceph-client.admin.57593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57391.log 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.48898.log.gz 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56324.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.57593.log.gz 2024-03-23T14:38:10.772 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44125.log 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.56324.log.gz 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49564.log 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57391.log: /var/log/ceph/ceph-client.admin.44125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50840.log 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.57391.log.gz 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49564.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.44125.log.gz 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43777.log 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.49564.log.gz 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57265.log 2024-03-23T14:38:10.773 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43777.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.50840.log.gz 2024-03-23T14:38:10.774 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54301.log 2024-03-23T14:38:10.774 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57265.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.43777.log.gz 2024-03-23T14:38:10.774 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55676.log 2024-03-23T14:38:10.774 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.57265.log.gz 2024-03-23T14:38:10.774 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54301.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56282.log 2024-03-23T14:38:10.774 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8%/var/log/ceph/ceph-client.admin.55676.log: -- replaced with /var/log/ceph/ceph-client.admin.54301.log.gz 2024-03-23T14:38:10.774 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60034.log 2024-03-23T14:38:10.774 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56282.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55676.log.gz 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49018.log 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.56282.log.gz 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51829.log 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49018.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.60034.log.gz 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.56996.log 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51829.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49018.log.gz 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51829.log.gz 2024-03-23T14:38:10.775 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.0.56996.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.45801.log 2024-03-23T14:38:10.776 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56622.log 2024-03-23T14:38:10.776 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.0.45801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51743.log 2024-03-23T14:38:10.776 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50714.log 2024-03-23T14:38:10.776 INFO:teuthology.orchestra.run.smithi161.stderr: 90.4%/var/log/ceph/ceph-client.admin.51743.log: 91.6% -- replaced with /var/log/ceph/ceph-client.0.56996.log.gz 2024-03-23T14:38:10.776 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58206.log 2024-03-23T14:38:10.776 INFO:teuthology.orchestra.run.smithi161.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.56622.log.gz 2024-03-23T14:38:10.776 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.0.45801.log.gz 2024-03-23T14:38:10.776 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.admin.50714.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.55382.log 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.51743.log.gz 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58206.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46867.log 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50714.log.gz 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.58206.log.gz 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54878.log 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55382.log.gz 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55172.log 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46867.log: /var/log/ceph/ceph-client.admin.54878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61142.log 2024-03-23T14:38:10.777 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46867.log.gz 2024-03-23T14:38:10.778 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55172.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54878.log.gz 2024-03-23T14:38:10.778 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44659.log 2024-03-23T14:38:10.778 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.55172.log.gz 2024-03-23T14:38:10.778 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45247.log 2024-03-23T14:38:10.778 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.61142.log.gz 2024-03-23T14:38:10.778 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44659.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51008.log 2024-03-23T14:38:10.778 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45247.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58432.log 2024-03-23T14:38:10.778 INFO:teuthology.orchestra.run.smithi161.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.44659.log.gz 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.45247.log.gz 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42866.log 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49648.log 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58432.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51008.log.gz 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42866.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58432.log.gz 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53069.log 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.42866.log.gz 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43142.log 2024-03-23T14:38:10.779 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49648.log: /var/log/ceph/ceph-client.admin.53069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47848.log 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49648.log.gz 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53069.log.gz 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43142.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48546.log 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.43142.log.gz 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47848.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43651.log 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.47848.log.gz 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44463.log 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48546.log: /var/log/ceph/ceph-client.admin.43651.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.48546.log.gz 2024-03-23T14:38:10.780 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44458.log 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43651.log.gz 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52483.log 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44463.log: /var/log/ceph/ceph-client.admin.44458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50756.log 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44463.log.gz 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44458.log.gz 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52131.log 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46421.log 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50756.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.52483.log.gz 2024-03-23T14:38:10.781 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52131.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.50756.log.gz 2024-03-23T14:38:10.782 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54175.log 2024-03-23T14:38:10.782 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52131.log.gz 2024-03-23T14:38:10.782 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61184.log 2024-03-23T14:38:10.782 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46421.log: /var/log/ceph/ceph-client.admin.54175.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44126.log 2024-03-23T14:38:10.782 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.46421.log.gz 2024-03-23T14:38:10.782 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54175.log.gz 2024-03-23T14:38:10.782 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61184.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44117.log 2024-03-23T14:38:10.783 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.61184.log.gz 2024-03-23T14:38:10.783 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44785.log 2024-03-23T14:38:10.783 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44126.log: /var/log/ceph/ceph-client.admin.44117.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44126.log.gz 2024-03-23T14:38:10.783 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59595.log 2024-03-23T14:38:10.783 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44117.log.gz 2024-03-23T14:38:10.783 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49732.log 2024-03-23T14:38:10.783 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44785.log: /var/log/ceph/ceph-client.admin.59595.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58148.log 2024-03-23T14:38:10.783 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44785.log.gz 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.59595.log.gz 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49732.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59469.log 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.49732.log.gz 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57307.log 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58148.log.gz 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59469.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43903.log 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.59469.log.gz 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50230.log 2024-03-23T14:38:10.784 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57307.log: /var/log/ceph/ceph-client.admin.43903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55975.log 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.57307.log.gz 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50230.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43903.log.gz 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43525.log 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.50230.log.gz 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55975.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49438.log 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.55975.log.gz 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54359.log 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43525.log.gz 2024-03-23T14:38:10.785 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53027.log 2024-03-23T14:38:10.786 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49438.log: /var/log/ceph/ceph-client.admin.54359.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60974.log 2024-03-23T14:38:10.786 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49438.log.gz 2024-03-23T14:38:10.786 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53027.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54359.log.gz 2024-03-23T14:38:10.786 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55298.log 2024-03-23T14:38:10.786 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53027.log.gz 2024-03-23T14:38:10.786 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60974.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56240.log 2024-03-23T14:38:10.786 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55298.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60974.log.gz 2024-03-23T14:38:10.786 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51134.log 2024-03-23T14:38:10.787 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.55298.log.gz 2024-03-23T14:38:10.787 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56240.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53505.log 2024-03-23T14:38:10.787 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.56240.log.gz 2024-03-23T14:38:10.787 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.31640.log 2024-03-23T14:38:10.787 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51134.log: /var/log/ceph/ceph-client.admin.53505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58474.log 2024-03-23T14:38:10.787 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.51134.log.gz 2024-03-23T14:38:10.787 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53505.log.gz 2024-03-23T14:38:10.787 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.tmp-client.admin.31640.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47622.log 2024-03-23T14:38:10.788 INFO:teuthology.orchestra.run.smithi161.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.31640.log.gz 2024-03-23T14:38:10.788 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58474.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44124.log 2024-03-23T14:38:10.788 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47622.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.58474.log.gz 2024-03-23T14:38:10.788 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44701.log 2024-03-23T14:38:10.788 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47622.log.gz 2024-03-23T14:38:10.788 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.0.53885.log 2024-03-23T14:38:10.788 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44124.log: /var/log/ceph/ceph-client.admin.44701.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-23T14:38:10.789 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44124.log.gz 2024-03-23T14:38:10.789 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.44701.log.gz 2024-03-23T14:38:10.789 INFO:teuthology.orchestra.run.smithi161.stderr:gzip/var/log/ceph/ceph-client.0.53885.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44461.log 2024-03-23T14:38:10.789 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46741.log 2024-03-23T14:38:10.789 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60459.log 2024-03-23T14:38:10.789 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44461.log.gz 2024-03-23T14:38:10.789 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46741.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51326.log 2024-03-23T14:38:10.789 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60459.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.46741.log.gz 2024-03-23T14:38:10.790 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55933.log 2024-03-23T14:38:10.790 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51326.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.60459.log.gz 2024-03-23T14:38:10.790 INFO:teuthology.orchestra.run.smithi161.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.0.53885.log.gz 2024-03-23T14:38:10.790 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49270.log 2024-03-23T14:38:10.790 INFO:teuthology.orchestra.run.smithi161.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.51326.log.gz 2024-03-23T14:38:10.790 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48772.log 2024-03-23T14:38:10.790 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55933.log: /var/log/ceph/ceph-client.admin.49270.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43861.log 2024-03-23T14:38:10.790 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55933.log.gz 2024-03-23T14:38:10.791 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49270.log.gz 2024-03-23T14:38:10.791 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48772.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57635.log 2024-03-23T14:38:10.791 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50356.log 2024-03-23T14:38:10.791 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43861.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48772.log.gz 2024-03-23T14:38:10.791 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57635.log: gzip -5 --verbose 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43861.log.gz 2024-03-23T14:38:10.791 INFO:teuthology.orchestra.run.smithi161.stderr: -- /var/log/ceph/ceph-client.admin.57551.log 2024-03-23T14:38:10.791 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50356.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.57635.log.gz 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53421.log 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.50356.log.gz 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54091.log 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57551.log: /var/log/ceph/ceph-client.admin.53421.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51050.log 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.57551.log.gz 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53421.log.gz 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54091.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59679.log 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48976.log 2024-03-23T14:38:10.792 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51050.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.54091.log.gz 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59679.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55340.log 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51050.log.gz 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48976.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.59679.log.gz 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55771.log 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48976.log.gz 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50798.log 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59191.log 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.55340.log.gz/var/log/ceph/ceph-client.admin.55771.log: 2024-03-23T14:38:10.793 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58374.log 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55771.log.gz 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59191.log: gzip 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50798.log.gz 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49816.log 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.59191.log.gz 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58374.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52173.log 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.58374.log.gz 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49816.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56664.log 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52173.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49816.log.gz 2024-03-23T14:38:10.794 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52173.log.gz 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr: /var/log/ceph/ceph-client.admin.60416.log 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58248.log 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60416.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48730.log 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.56664.log.gz 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58248.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.60416.log.gz 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50882.log 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.58248.log.gz 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61016.log 2024-03-23T14:38:10.795 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50882.log: gzip -5 --verbose 88.8% -- replaced with /var/log/ceph/ceph-client.admin.48730.log.gz 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr: -- /var/log/ceph/ceph-client.0.51586.log 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50882.log.gz 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61016.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52399.log 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.61016.log.gz 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.0.51586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59233.log 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44995.log 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52399.log.gz 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59233.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49144.log 2024-03-23T14:38:10.796 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.59233.log.gz 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48304.log 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49144.log: gzip -5 --verbose 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44995.log.gz 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr: -- /var/log/ceph/ceph-client.admin.46909.log 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8%/var/log/ceph/ceph-client.admin.48304.log: -- replaced with /var/log/ceph/ceph-client.admin.49144.log.gz 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.0.51586.log.gz 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59275.log 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48304.log.gz 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-23T14:38:10.797 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46909.log: /var/log/ceph/ceph-client.admin.59275.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52005.log 2024-03-23T14:38:10.798 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46909.log.gz 2024-03-23T14:38:10.798 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mgr.y.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.59275.log.gz 2024-03-23T14:38:10.798 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59889.log 2024-03-23T14:38:10.798 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47748.log 2024-03-23T14:38:10.798 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59889.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52005.log.gz 2024-03-23T14:38:10.798 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60501.log 2024-03-23T14:38:10.798 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47748.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49606.log 2024-03-23T14:38:10.798 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.59889.log.gz 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51787.log 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47748.log.gz/var/log/ceph/ceph-client.admin.60501.log: 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49606.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50524.log 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.60501.log.gz 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50022.log 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49606.log.gz 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51787.log: /var/log/ceph/ceph-client.admin.50524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55592.log 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50022.log: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48094.log 2024-03-23T14:38:10.799 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.50524.log.gz 2024-03-23T14:38:10.800 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.51787.log.gz 2024-03-23T14:38:10.800 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55592.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.50022.log.gz 2024-03-23T14:38:10.800 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44953.log 2024-03-23T14:38:10.800 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.55592.log.gz 2024-03-23T14:38:10.800 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49858.log 2024-03-23T14:38:10.800 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48094.log: /var/log/ceph/ceph-client.admin.44953.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50612.log 2024-03-23T14:38:10.800 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48094.log.gz 2024-03-23T14:38:10.800 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6%/var/log/ceph/ceph-client.admin.49858.log: -- replaced with /var/log/ceph/ceph-client.admin.44953.log.gz 2024-03-23T14:38:10.801 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57165.log 2024-03-23T14:38:10.801 INFO:teuthology.orchestra.run.smithi161.stderr: 93.0% 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49858.log.gz 2024-03-23T14:38:10.801 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-23T14:38:10.801 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49900.log 2024-03-23T14:38:10.801 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57165.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.50612.log.gz 2024-03-23T14:38:10.801 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57223.log 2024-03-23T14:38:10.801 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57165.log.gz 2024-03-23T14:38:10.801 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56059.log 2024-03-23T14:38:10.802 INFO:teuthology.orchestra.run.smithi161.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49900.log.gz 2024-03-23T14:38:10.802 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.57223.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51947.log 2024-03-23T14:38:10.802 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47890.log 2024-03-23T14:38:10.802 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.57223.log.gz 2024-03-23T14:38:10.802 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56059.log: /var/log/ceph/ceph-client.admin.51947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59511.log 2024-03-23T14:38:10.802 INFO:teuthology.orchestra.run.smithi161.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.56059.log.gz 2024-03-23T14:38:10.802 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47890.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51947.log.gz 2024-03-23T14:38:10.802 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52273.log 2024-03-23T14:38:10.803 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47890.log.gz 2024-03-23T14:38:10.803 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59511.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48430.log 2024-03-23T14:38:10.803 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47119.log 2024-03-23T14:38:10.803 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.59511.log.gz 2024-03-23T14:38:10.803 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52273.log: /var/log/ceph/ceph-client.admin.48430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47443.log 2024-03-23T14:38:10.803 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52273.log.gz 2024-03-23T14:38:10.803 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47119.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.48430.log.gz 2024-03-23T14:38:10.803 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56580.log 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47119.log.gz 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47443.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47664.log 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60551.log 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.47443.log.gz 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56580.log: /var/log/ceph/ceph-client.admin.47664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55130.log 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.56580.log.gz 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.47664.log.gz 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59317.log 2024-03-23T14:38:10.804 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55130.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48814.log 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.60551.log.gz 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.55130.log.gz 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59317.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.2.46299.log 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48814.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.59317.log.gz 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.2.46299.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48220.log 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.48814.log.gz 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47077.log 2024-03-23T14:38:10.805 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48220.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43693.log 2024-03-23T14:38:10.806 INFO:teuthology.orchestra.run.smithi161.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.2.46299.log.gz/var/log/ceph/ceph-client.admin.47077.log: 2024-03-23T14:38:10.806 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48220.log.gz 2024-03-23T14:38:10.806 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56538.log 2024-03-23T14:38:10.806 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.47077.log.gz 2024-03-23T14:38:10.806 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42947.log 2024-03-23T14:38:10.806 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56538.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43693.log.gz 2024-03-23T14:38:10.806 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47225.log 2024-03-23T14:38:10.806 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56538.log.gz 2024-03-23T14:38:10.807 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.42947.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58558.log 2024-03-23T14:38:10.807 INFO:teuthology.orchestra.run.smithi161.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42947.log.gz 2024-03-23T14:38:10.807 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47225.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53111.log 2024-03-23T14:38:10.807 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60848.log 2024-03-23T14:38:10.807 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.47225.log.gz 2024-03-23T14:38:10.807 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53111.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43353.log 2024-03-23T14:38:10.807 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.58558.log.gz 2024-03-23T14:38:10.807 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53111.log.gz 2024-03-23T14:38:10.808 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60848.log: -- /var/log/ceph/ceph-client.admin.47932.log 2024-03-23T14:38:10.808 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43353.log: 87.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45163.log 2024-03-23T14:38:10.808 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.60848.log.gz 2024-03-23T14:38:10.808 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.43353.log.gz 2024-03-23T14:38:10.808 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47932.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-23T14:38:10.808 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45163.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45037.log 2024-03-23T14:38:10.808 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47932.log.gz 2024-03-23T14:38:10.808 INFO:teuthology.orchestra.run.smithi161.stderr: 88.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44123.log 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph.log: -- replaced with /var/log/ceph/ceph-client.admin.45163.log.gz 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45037.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54920.log 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr:gzip 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45037.log.gz 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr: -5 --verbose/var/log/ceph/ceph-client.admin.44123.log: -- /var/log/ceph/ceph-osd.2.log 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr: 85.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54527.log 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.44123.log.gz/var/log/ceph/ceph-client.admin.54920.log: 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose/var/log/ceph/ceph-osd.2.log: -- /var/log/ceph/ceph-mds.c.log 2024-03-23T14:38:10.809 INFO:teuthology.orchestra.run.smithi161.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54920.log.gz 2024-03-23T14:38:10.810 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54527.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53583.log 2024-03-23T14:38:10.810 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.54527.log.gz/var/log/ceph/ceph-mds.c.log: 2024-03-23T14:38:10.810 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53195.log 2024-03-23T14:38:10.810 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43439.log 2024-03-23T14:38:10.810 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53583.log: /var/log/ceph/ceph-client.admin.53195.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.53583.log.gz 2024-03-23T14:38:10.810 INFO:teuthology.orchestra.run.smithi161.stderr: 92.6% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 88.2% -- replaced with /var/log/ceph/ceph-client.admin.53195.log.gz 2024-03-23T14:38:10.810 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-23T14:38:10.810 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45289.log 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.43439.log: -- /var/log/ceph/ceph-client.admin.50187.log 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45289.log: 88.0%gzip -- replaced with /var/log/ceph/ceph-client.admin.43439.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.37953.log 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55256.log 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50187.log: /var/log/ceph/ceph-client.admin.37953.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.45289.log.gz 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50187.log.gz 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.37953.log.gz 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44452.log 2024-03-23T14:38:10.811 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55256.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48136.log 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55256.log.gz 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49228.log 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44452.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54443.log 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48136.log: /var/log/ceph/ceph-client.admin.49228.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48136.log.gz 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.49228.log.gz 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44743.log 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48630.log 2024-03-23T14:38:10.812 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54443.log: /var/log/ceph/ceph-client.admin.44743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.57973.log 2024-03-23T14:38:10.813 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48630.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.54443.log.gz 2024-03-23T14:38:10.813 INFO:teuthology.orchestra.run.smithi161.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44743.log.gz 2024-03-23T14:38:10.813 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43311.log 2024-03-23T14:38:10.813 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.48630.log.gz 2024-03-23T14:38:10.813 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46657.log 2024-03-23T14:38:10.813 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.0.57973.log: /var/log/ceph/ceph-client.admin.43311.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48178.log 2024-03-23T14:38:10.813 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46657.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.43311.log.gz 2024-03-23T14:38:10.813 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.46657.log.gz 2024-03-23T14:38:10.814 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54569.log 2024-03-23T14:38:10.814 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.48178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54665.log 2024-03-23T14:38:10.814 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54569.log: 87.8% 90.0% -- replaced with /var/log/ceph/ceph-client.0.57973.log.gz 2024-03-23T14:38:10.814 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.48178.log.gz 2024-03-23T14:38:10.814 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.54569.log.gz 2024-03-23T14:38:10.814 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49942.log 2024-03-23T14:38:10.814 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55550.log 2024-03-23T14:38:10.814 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.54665.log: /var/log/ceph/ceph-client.admin.49942.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55424.log 2024-03-23T14:38:10.815 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55550.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.54665.log.gz 2024-03-23T14:38:10.815 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49942.log.gz 2024-03-23T14:38:10.815 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57509.log 2024-03-23T14:38:10.815 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.55550.log.gz 2024-03-23T14:38:10.815 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43567.log 2024-03-23T14:38:10.815 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.55424.log: /var/log/ceph/ceph-client.admin.57509.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47035.log 2024-03-23T14:38:10.815 INFO:teuthology.orchestra.run.smithi161.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.55424.log.gz 2024-03-23T14:38:10.815 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43567.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.57509.log.gz 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38072.log 87.8% -- replaced with /var/log/ceph/ceph-client.admin.43567.log.gz 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51092.log 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.47035.log.gz 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.38072.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53379.log 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38072.log.gz 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.51092.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53625.log 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.51092.log.gz 2024-03-23T14:38:10.816 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44911.log 2024-03-23T14:38:10.817 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.53379.log: /var/log/ceph/ceph-client.admin.53625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49354.log 2024-03-23T14:38:10.817 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44911.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.53379.log.gz 2024-03-23T14:38:10.817 INFO:teuthology.orchestra.run.smithi161.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.53625.log.gz 2024-03-23T14:38:10.817 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56496.log 2024-03-23T14:38:10.817 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.44911.log.gz 2024-03-23T14:38:10.817 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50966.log 2024-03-23T14:38:10.817 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49354.log: /var/log/ceph/ceph-client.admin.56496.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60722.log 2024-03-23T14:38:10.817 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.50966.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.49354.log.gz 2024-03-23T14:38:10.818 INFO:teuthology.orchestra.run.smithi161.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.56496.log.gz 2024-03-23T14:38:10.818 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44457.log 2024-03-23T14:38:10.818 INFO:teuthology.orchestra.run.smithi161.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.50966.log.gz 2024-03-23T14:38:10.818 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47580.log 2024-03-23T14:38:10.818 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60722.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59112.log 2024-03-23T14:38:10.818 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44457.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.60722.log.gz 2024-03-23T14:38:10.818 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.47580.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44457.log.gz 2024-03-23T14:38:10.818 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52089.log 2024-03-23T14:38:10.819 INFO:teuthology.orchestra.run.smithi161.stderr: -- replaced with /var/log/ceph/ceph-client.admin.47580.log.gz 2024-03-23T14:38:10.819 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.59112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54794.log 2024-03-23T14:38:10.819 INFO:teuthology.orchestra.run.smithi161.stderr:gzip 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59112.log.gz -5 2024-03-23T14:38:10.819 INFO:teuthology.orchestra.run.smithi161.stderr: --verbose -- /var/log/ceph/ceph-client.admin.49480.log 2024-03-23T14:38:10.819 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.52089.log: /var/log/ceph/ceph-client.admin.54794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43945.log 2024-03-23T14:38:10.819 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49480.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.52089.log.gz 2024-03-23T14:38:10.819 INFO:teuthology.orchestra.run.smithi161.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.54794.log.gz 2024-03-23T14:38:10.819 INFO:teuthology.orchestra.run.smithi161.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49480.log.gz 2024-03-23T14:38:10.820 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60679.log 2024-03-23T14:38:10.820 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.43945.log: -- /var/log/ceph/ceph-client.admin.49690.log 2024-03-23T14:38:10.820 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60679.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.43945.log.gz 2024-03-23T14:38:10.820 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54133.log 2024-03-23T14:38:10.820 INFO:teuthology.orchestra.run.smithi161.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.60679.log.gz 2024-03-23T14:38:10.820 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.a.log 2024-03-23T14:38:10.820 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.49690.log: /var/log/ceph/ceph-client.admin.54133.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49690.log.gz 2024-03-23T14:38:10.820 INFO:teuthology.orchestra.run.smithi161.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54133.log.gz 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53153.log 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46993.log 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-mds.a.log: /var/log/ceph/ceph-client.admin.53153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.60136.log 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.46993.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53153.log.gz 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.46993.log.gz 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44869.log 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.60136.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58332.log 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.44869.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.60136.log.gz 2024-03-23T14:38:10.821 INFO:teuthology.orchestra.run.smithi161.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44869.log.gz 2024-03-23T14:38:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43483.log 2024-03-23T14:38:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.58332.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.58332.log.gz 2024-03-23T14:38:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56145.log 2024-03-23T14:38:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.43483.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45923.log 2024-03-23T14:38:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.56145.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43483.log.gz 2024-03-23T14:38:10.822 INFO:teuthology.orchestra.run.smithi161.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.56145.log.gz 2024-03-23T14:38:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.61100.log 2024-03-23T14:38:10.822 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.45923.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.45923.log.gz 2024-03-23T14:38:10.823 INFO:teuthology.orchestra.run.smithi161.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.47974.log 2024-03-23T14:38:10.823 INFO:teuthology.orchestra.run.smithi161.stderr:/var/log/ceph/ceph-client.admin.61100.log: /var/log/ceph/ceph-client.admin.47974.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.61100.log.gz 2024-03-23T14:38:10.823 INFO:teuthology.orchestra.run.smithi161.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.47974.log.gz 2024-03-23T14:38:10.823 INFO:teuthology.orchestra.run.smithi161.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-03-23T14:38:10.824 INFO:teuthology.orchestra.run.smithi161.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-23T14:38:10.824 INFO:teuthology.orchestra.run.smithi183.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-23T14:38:10.824 INFO:teuthology.orchestra.run.smithi183.stderr: 93.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-23T14:38:10.824 INFO:teuthology.orchestra.run.smithi183.stderr: 91.1% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-23T14:38:10.825 INFO:teuthology.orchestra.run.smithi183.stderr: 93.5% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-23T14:38:10.825 INFO:teuthology.orchestra.run.smithi183.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-23T14:38:10.826 INFO:teuthology.orchestra.run.smithi183.stderr: 93.8% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-23T14:38:10.828 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-23T14:38:10.828 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m0.228s 2024-03-23T14:38:10.829 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m1.134s 2024-03-23T14:38:10.829 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.070s 2024-03-23T14:38:10.829 INFO:teuthology.orchestra.run.smithi161.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-23T14:38:10.838 INFO:teuthology.orchestra.run.smithi161.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-23T14:38:10.845 INFO:teuthology.orchestra.run.smithi161.stderr: 93.7% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-23T14:38:10.888 INFO:teuthology.orchestra.run.smithi161.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-23T14:38:10.891 INFO:teuthology.orchestra.run.smithi161.stderr: 2024-03-23T14:38:10.891 INFO:teuthology.orchestra.run.smithi161.stderr:real 0m0.314s 2024-03-23T14:38:10.891 INFO:teuthology.orchestra.run.smithi161.stderr:user 0m1.113s 2024-03-23T14:38:10.891 INFO:teuthology.orchestra.run.smithi161.stderr:sys 0m0.253s 2024-03-23T14:38:10.891 INFO:tasks.ceph:Archiving logs... 2024-03-23T14:38:10.892 DEBUG:teuthology.misc:Transferring archived files from smithi161:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/remote/smithi161/log 2024-03-23T14:38:10.893 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T14:38:11.357 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/remote/smithi183/log 2024-03-23T14:38:11.357 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-23T14:38:11.531 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-23T14:38:11.556 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-23T14:38:11.556 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T14:38:11.559 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-23T14:38:11.633 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-23T14:38:11.645 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-23T14:38:11.645 DEBUG:teuthology.orchestra.run.smithi161:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T14:38:11.648 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-23T14:38:11.660 INFO:teuthology.orchestra.run.smithi161.stderr:bash: line 1: ntpq: command not found 2024-03-23T14:38:11.662 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-23T14:38:11.665 INFO:teuthology.orchestra.run.smithi161.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T14:38:11.665 INFO:teuthology.orchestra.run.smithi161.stdout:=============================================================================== 2024-03-23T14:38:11.665 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +1269us[+1269us] +/- 98ms 2024-03-23T14:38:11.665 INFO:teuthology.orchestra.run.smithi161.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -423us[ -290us] +/- 64ms 2024-03-23T14:38:11.665 INFO:teuthology.orchestra.run.smithi161.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -390us[ -390us] +/- 74ms 2024-03-23T14:38:11.665 INFO:teuthology.orchestra.run.smithi161.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:38:11.667 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-23T14:38:11.667 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-23T14:38:11.667 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 33 +1265us[+1265us] +/- 98ms 2024-03-23T14:38:11.667 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -424us[ -294us] +/- 64ms 2024-03-23T14:38:11.668 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 34 -374us[ -245us] +/- 74ms 2024-03-23T14:38:11.668 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-23T14:38:11.668 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-23T14:38:11.686 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-23T14:38:11.686 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-23T14:38:11.737 DEBUG:teuthology.orchestra.run.smithi161:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T14:38:11.816 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-23T14:38:11.884 DEBUG:teuthology.orchestra.run.smithi161:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T14:38:11.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:38:11.914 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T14:38:11.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:38:11.944 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-23T14:38:11.956 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-23T14:38:11.983 INFO:teuthology.task.internal:Duration was 618.832233 seconds 2024-03-23T14:38:11.984 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-23T14:38:12.033 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-23T14:38:12.033 DEBUG:teuthology.orchestra.run.smithi161:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T14:38:12.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-23T14:38:12.077 INFO:teuthology.orchestra.run.smithi161.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T14:38:12.079 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T14:38:12.406 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-23T14:38:12.406 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi161.front.sepia.ceph.com 2024-03-23T14:38:12.406 DEBUG:teuthology.orchestra.run.smithi161:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T14:38:12.467 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-23T14:38:12.467 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-23T14:38:12.495 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-23T14:38:12.495 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T14:38:12.510 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-23T14:38:12.563 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-23T14:38:12.564 DEBUG:teuthology.orchestra.run.smithi161:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T14:38:12.578 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-23T14:38:12.704 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-23T14:38:12.714 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-23T14:38:12.714 DEBUG:teuthology.orchestra.run.smithi161:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T14:38:12.741 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-23T14:38:12.768 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-23T14:38:12.781 DEBUG:teuthology.orchestra.run.smithi161:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T14:38:12.784 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-23T14:38:12.807 INFO:teuthology.orchestra.run.smithi161.stdout:kernel.core_pattern = core 2024-03-23T14:38:12.833 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-23T14:38:12.849 DEBUG:teuthology.orchestra.run.smithi161:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T14:38:12.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:38:12.878 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-23T14:38:12.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T14:38:12.903 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-23T14:38:12.913 INFO:teuthology.task.internal:Transferring archived files... 2024-03-23T14:38:12.914 DEBUG:teuthology.misc:Transferring archived files from smithi161:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/remote/smithi161 2024-03-23T14:38:12.914 DEBUG:teuthology.orchestra.run.smithi161:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T14:38:12.954 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384/remote/smithi183 2024-03-23T14:38:12.955 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-23T14:38:12.989 INFO:teuthology.task.internal:Removing archive directory... 2024-03-23T14:38:12.989 DEBUG:teuthology.orchestra.run.smithi161:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T14:38:12.992 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-23T14:38:13.039 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-23T14:38:13.051 INFO:teuthology.task.internal:Not uploading archives. 2024-03-23T14:38:13.051 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-23T14:38:13.078 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-23T14:38:13.078 DEBUG:teuthology.orchestra.run.smithi161:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T14:38:13.081 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-23T14:38:13.093 INFO:teuthology.orchestra.run.smithi161.stdout: 265523 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 14:38 /home/ubuntu/cephtest 2024-03-23T14:38:13.095 INFO:teuthology.orchestra.run.smithi183.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 23 14:38 /home/ubuntu/cephtest 2024-03-23T14:38:13.096 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-23T14:38:13.129 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-23T14:38:13.205 INFO:teuthology.nuke:Checking targets against current locks 2024-03-23T14:38:13.234 DEBUG:teuthology.nuke:shortname: smithi161 2024-03-23T14:38:13.234 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T14:38:13.257 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-23T14:38:13.257 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T14:38:13.261 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi161.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 14:19:37.511253', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFKdulghjUx2dBVzfkJOysFyenP9PDwo02ofmLKEojk9EStLnapx7pnDP8GFdWOec+snZKV2BIzwqhOjM9JWDjM='} 2024-03-23T14:38:13.286 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-03-23_09:56:53-fs-wip-batrick-testing-20240323.003144-squid-distro-default-smithi/7618384', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 14:19:37.509919', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIC7RNbUBA5eX59Y7zf4jpLUGwBV7jKzulhCtUEFQOIZwAED998+oGq2kxB7gy9IynqcatepIhLGMJoYIJxvDj0='} 2024-03-23T14:38:13.316 INFO:teuthology.orchestra.console.smithi161:Power off 2024-03-23T14:38:13.317 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T14:38:13.339 DEBUG:teuthology.orchestra.console.smithi161:power off output: Chassis Power Control: Down/Off 2024-03-23T14:38:13.340 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T14:38:13.358 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-23T14:38:13.359 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-23T14:38:13.368 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-23T14:38:13.381 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-23T14:38:13.381 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T14:38:13.404 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-23T14:38:17.369 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T14:38:17.405 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T14:38:17.479 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is on 2024-03-23T14:38:17.516 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-23T14:38:21.484 DEBUG:teuthology.orchestra.console.smithi161:pexpect command: ipmitool -H smithi161.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T14:38:21.517 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T14:38:21.595 DEBUG:teuthology.orchestra.console.smithi161:check power output: Chassis Power is off 2024-03-23T14:38:21.627 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-23T14:38:21.696 INFO:teuthology.orchestra.console.smithi161:Power off completed 2024-03-23T14:38:25.630 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-23T14:38:25.740 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-23T14:38:25.840 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-23T14:38:25.945 INFO:teuthology.run:Summary data: description: fs/functional/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a3s-mds-4c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse objectstore/bluestore-bitmap overrides/{ignorelist_health ignorelist_wrongly_marked_down no_client_pidfile pg_health} subvol_versions/create_subvol_version_v1 tasks/pool-perm} duration: 618.8322329521179 failure_reason: 'Test failure: test_pool_perm (tasks.cephfs.test_pool_perm.TestPoolPerm)' flavor: default owner: scheduled_pdonnell@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=f278c6e9690a4392ae3c720a3818398d status: fail success: false 2024-03-23T14:38:25.945 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T14:38:26.031 INFO:teuthology.run:FAIL